AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversaries
Authors:
Preprint
Abstract
Contrastive learning relies on constructing a collection of negative examples that are sufficiently hard to discriminate against positive queries when their representations are self-trained. Existing contrastive learning methods either maintain a queue of negative samples over minibatches while only a small portion of them are updated in an iteration, or only use the other examples from the current minibatch as negatives. They could not closely track the change of the learned representation over iterations by updating the entire queue as a whole, or discard the useful information from the past minibatches. Alternatively, we present to directly learn a set of negative adversaries playing against the self-trained representation. Two players, the representation network and negative adversaries, are alternately updated to obtain the most challenging negative examples against which the representation of positive queries will be trained to discriminate. We further show that the negative adversaries are updated towards a weighted combination of positive queries by maximizing the adversarial contrastive loss, thereby allowing them to closely track the change of representations over time. Experiment results demonstrate the proposed Adversarial Contrastive (AdCo) model not only achieves superior performances (a top-1 accuracy of 73.2\% over 200 epochs and 75.7\% over 800 epochs with linear evaluation on ImageNet), but also can be pre-trained more efficiently with fewer epochs.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Exploring Simple Siamese Representation Learning2021
- Contrastive Multiview Coding2020
- Small Data Challenges in Big Data Era: A Survey of Recent Progress on Unsupervised and Semi-Supervised MethodsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
- Learning Generalized Transformation Equivariant Representations Via AutoEncoding TransformationsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
- Momentum Contrast for Unsupervised Visual Representation Learning2020
- Loss-Sensitive Generative Adversarial Networks on Lipschitz DensitiesInternational Journal of Computer Vision, 2019
- AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transformations Rather Than Data2019