Addressing Feature Suppression in Unsupervised Visual Representations
Authors:
- 1 January 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1411-1420
Abstract
Contrastive learning is one of the fastest growing research areas in machine learning due to its ability to learn useful representations without labeled data. However, contrastive learning is susceptible to feature suppression – i.e., it may discard important information relevant to the task of interest, and learn irrelevant features. Past work has addressed this limitation via handcrafted data augmentations that eliminate irrelevant information. This approach however does not work across all datasets and tasks. Further, data augmentations fail in addressing feature suppression in multi-attribute classification when one attribute can suppress features relevant to other attributes. In this paper, we analyze the objective function of contrastive learning and formally prove that it is vulnerable to feature suppression. We then present Predictive Contrastive Learning (PrCL), a framework for learning unsupervised representations that are robust to feature suppression. The key idea is to force the learned representation to predict the input, and hence prevent it from discarding important information. Extensive experiments verify that PrCL is robust to feature suppression and outperforms state-of-the-art contrastive learning methods on a variety of datasets and tasks.Keywords
All Related Versions
This publication has 30 references indexed in Scilit:
- Controllable Image-to-Video Translation: A Case Study on Facial Expression GenerationProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Unsupervised Embedding Learning via Invariant and Spreading Instance Feature2019
- Toward Efficient Action Recognition: Principal Backpropagation for Training Two-Stream NetworksIEEE Transactions on Image Processing, 2018
- End-to-End Learning of Motion Representation for Video Understanding2018
- Context Encoders: Feature Learning by Inpainting2016
- Unsupervised Visual Representation Learning by Context Prediction2015
- 2D Human Pose Estimation: New Benchmark and State of the Art Analysis2014
- ImageNet: A large-scale hierarchical image database2009
- Extracting and composing robust features with denoising autoencoders2008
- Reducing the Dimensionality of Data with Neural NetworksScience, 2006