Information-Preserving Contrastive Learning for Self-Supervised Representations
Preprint
Abstract
Contrastive learning is very effective at learning useful representations without supervision. Yet contrastive learning has its limitations. It may learn a shortcut that is irrelevant to the downstream task, and discard relevant information. Past work has addressed this limitation via custom data augmentations that eliminate the shortcut. This solution however does not work for data modalities that are not interpretable by humans, e.g., radio signals. For such modalities, it is hard for a human to guess which shortcuts may exist in the signal, or how they can be eliminated. Even for interpretable data, sometimes eliminating the shortcut may be undesirable. The shortcut may be irrelevant to one downstream task but important to another. In this case, it is desirable to learn a representation that captures both the shortcut information and the information relevant to the other downstream task. This paper presents information-preserving contrastive learning (IPCL), a new framework for unsupervised representation learning that preserves relevant information even in the presence of shortcuts. We empirically show that the representations learned by IPCL outperforms contrastive learning in supporting different modalities and multiple diverse downstream tasks.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age for Bias Measurement and Mitigation2021
- Memory-Augmented Dense Predictive Coding for Video Representation Learning2020
- Contrastive Multiview Coding2020
- In-Home Daily-Life Captioning Using Radio Signals2020
- Latern: Dynamic Continuous Hand Gesture Recognition Using FMCW Radar SensorIEEE Sensors Journal, 2018
- Zero-Effort In-Home Sleep and Insomnia Monitoring using Radio SignalsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2017