Transitive Invariance for Self-supervised Visual Representation Learning
Authors:
Preprint
Abstract
Learning visual representations with self-supervised learning has become popular in computer vision. The idea is to design auxiliary tasks where labels are free to obtain. Most of these tasks end up providing data to learn specific kinds of invariance useful for recognition. In this paper, we propose to exploit different self-supervised approaches to learn representations invariant to (i) inter-instance variations (two objects in the same class should have similar features) and (ii) intra-instance variations (viewpoint, pose, deformations, illumination, etc). Instead of combining two approaches with multi-task learning, we argue to organize and reason the data with multiple variations. Specifically, we propose to generate a graph with millions of objects mined from hundreds of thousands of videos. The objects are connected by two types of edges which correspond to two types of invariance: "different instances but a similar viewpoint and category" and "different viewpoints of the same instance". By applying simple transitivity on the graph with these edges, we can obtain pairs of images exhibiting richer visual invariance. We use this data to train a Triplet-Siamese network with VGG16 as the base architecture and apply the learned representations to different recognition tasks. For object detection, we achieve 63.2% mAP on PASCAL VOC 2007 using Fast R-CNN (compare to 67.3% with ImageNet pre-training). For the challenging COCO dataset, our method is surprisingly close (23.5%) to the ImageNet-supervised counterpart (24.4%) using the Faster R-CNN framework. We also show that our network can perform significantly better than the ImageNet network in the surface normal estimation task.Keywords
All Related Versions
This publication has 41 references indexed in Scilit:
- Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification2016
- Deep Metric Learning Using Triplet Network2015
- Discriminatively Trained Dense Surface Normal Estimation2014
- Unfolding an Indoor Origami World2014
- Context as Supervisory Signal: Discovering Objects with Predictable Context2014
- The Shape Boltzmann Machine: A Strong Model of Object ShapeInternational Journal of Computer Vision, 2013
- Selective Search for Object RecognitionInternational Journal of Computer Vision, 2013
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009
- Histograms of Oriented Gradients for Human Detection2005
- Distinctive Image Features from Scale-Invariant KeypointsInternational Journal of Computer Vision, 2004