i-Mix: A Strategy for Regularizing Contrastive Representation Learning
Preprint
Abstract
Contrastive representation learning has shown to be an effective way of learning representations from unlabeled data. However, much progress has been made in vision domains relying on data augmentations carefully designed using domain knowledge. In this work, we propose i-Mix, a simple yet effective regularization strategy for improving contrastive representation learning in both vision and non-vision domains. We cast contrastive learning as training a non-parametric classifier by assigning a unique virtual class to each data in a batch. Then, data instances are mixed in both the input and virtual label spaces, providing more augmented data during training. In experiments, we demonstrate that i-Mix consistently improves the quality of self-supervised representations across domains, resulting in significant performance gains on downstream tasks. Furthermore, we confirm its regularization effect via extensive ablation studies across model and dataset sizes.Keywords
All Related Versions
This publication has 33 references indexed in Scilit:
- Contrastive Multiview Coding2020
- Deep Clustering for Unsupervised Learning of Visual Features2018
- Audio-Visual Scene Analysis with Self-Supervised Multisensory Features2018
- Boosting Self-Supervised Learning via Knowledge Transfer2018
- Unsupervised Feature Learning via Non-parametric Instance Discrimination2018
- Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles2016
- Context Encoders: Feature Learning by Inpainting2016
- Deep Residual Learning for Image Recognition2016
- Rethinking the Inception Architecture for Computer Vision2016
- Going deeper with convolutions2015