Why Do Self-Supervised Models Transfer? Investigating the Impact of Invariance on Downstream Tasks
Preprint
Abstract
Self-supervised learning is a powerful paradigm for representation learning on unlabelled images. A wealth of effective new methods based on instance matching rely on data-augmentation to drive learning, and these have reached a rough agreement on an augmentation scheme that optimises popular recognition benchmarks. However, there is strong reason to suspect that different tasks in computer vision require features to encode different (in)variances, and therefore likely require different augmentation strategies. In this paper, we measure the invariances learned by contrastive methods and confirm that they do learn invariance to the augmentations used and further show that this invariance largely transfers to related real-world changes in pose and lighting. We show that learned invariances strongly affect downstream task performance and confirm that different downstream tasks benefit from polar opposite (in)variances, leading to performance loss when the standard augmentation strategy is used. Finally, we demonstrate that a simple fusion of representations with complementary invariances ensures wide transferability to all the diverse downstream tasks considered.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- How Well Do Self-Supervised Models Transfer?2021
- Benchmarking Representation Learning for Natural World Image Collections2021
- Learning Multi-Scale Photo Exposure Correction2021
- Real-World Blur Dataset for Learning and Benchmarking Deblurring Algorithms2020
- Momentum Contrast for Unsupervised Visual Representation Learning2020
- Using the Properties of Primate Motion Sensitive Neurons to Extract Camera Motion and Depth from Brief 2-D Monocular Image Sequences2019
- Deep Residual Learning for Image Recognition2016
- Microsoft COCO: Common Objects in Context2014
- Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categoriesComputer Vision and Image Understanding, 2007
- The Amsterdam Library of Object ImagesInternational Journal of Computer Vision, 2005