EMCLR: Expectation Maximization Contrastive Learning Representations
- 4 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-5
Abstract
One of the bottlenecks of self-supervised contrastive learning is the degenerate constant solution, where all the samples are mapped to one single point in representation space. To prevent such collapses, the mainstream paradigm is using negative samples, forcing negative pairs to push away. However, such manner results in ${\mathcal{O}}\left({{N^2}}\right)$ time and space complexities, limiting the expansibility, scalability and efficiency. We observe current negative-requiring objectives can be decomposed to alignment and uniformity, where uniformity dominates the ${\mathcal{O}}\left({{N^2}}\right)$ complexity. To reduce the complexity, inspired by the traditional EM algorithm, we derive the embedding matrix of each batch with optimally uniform distribution and discard the uniformity part in objectives. Specifically, for stacked embedding matrices of two views, we first calculate the optimal solution of one view by the proposed algorithm. Then we align the embedding matrix with the obtained optimal solution. The learning paradigm ingeniously avoids model collapses without ad-hoc negative pairs and reduces the square complexity to linear. Extensive experiments on CIFAR-10/100 and STL-10 show the proposed methods achieve comparable results in ${\mathcal{O}}\left(N\right)$ complexity.
Keywords
Funding Information
- National Key Research and Development Program of China
- National Natural Science Foundation of China
- Tencent
This publication has 9 references indexed in Scilit:
- On the Importance of Asymmetry for Siamese Representation Learning2022
- Align Representations with Base: A New Approach to Self-Supervised Learning2022
- On Feature Decorrelation in Self-Supervised Learning2021
- Emerging Properties in Self-Supervised Vision Transformers2021
- Exploring Simple Siamese Representation Learning2021
- Momentum Contrast for Unsupervised Visual Representation Learning2020
- Self-Supervised Representation Learning by Rotation Feature Decoupling2019
- Deep Residual Learning for Image Recognition2016
- Deep learningNature, 2015