i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning
Preprint
Abstract
Contrastive representation learning has shown to be effective to learn representations from unlabeled data. However, much progress has been made in vision domains relying on data augmentations carefully designed using domain knowledge. In this work, we propose i-Mix, a simple yet effective domain-agnostic regularization strategy for improving contrastive representation learning. We cast contrastive learning as training a non-parametric classifier by assigning a unique virtual class to each data in a batch. Then, data instances are mixed in both the input and virtual label spaces, providing more augmented data during training. In experiments, we demonstrate that i-Mix consistently improves the quality of learned representations across domains, including image, speech, and tabular data. Furthermore, we confirm its regularization effect via extensive ablation studies across model and dataset sizes. The code is available at https://github.com/kibok90/imix.Keywords
All Related Versions
This publication has 35 references indexed in Scilit:
- Contrastive Multiview Coding2020
- Comparing to Learn: Surpassing ImageNet Pretraining on Radiographs by Comparing Image Representations2020
- Deep Clustering for Unsupervised Learning of Visual Features2018
- Audio-Visual Scene Analysis with Self-Supervised Multisensory Features2018
- Unsupervised Feature Learning via Non-parametric Instance Discrimination2018
- Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles2016
- Context Encoders: Feature Learning by Inpainting2016
- Deep Residual Learning for Image Recognition2016
- Rethinking the Inception Architecture for Computer Vision2016
- Going deeper with convolutions2015