Understanding the Dimensional Need of Noncontrastive Learning
Top Cited Papers
- 3 July 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cybernetics
- Vol. 55 (9) , 4089-4102
Abstract
Noncontrastive self-supervised learning methods offer an effective alternative to contrastive approaches by avoiding the need for negative samples to avoid representation collapse. Noncontrastive learning methods explicitly or implicitly optimize the representation space, yet they often require large representation dimensions, leading to dimensional inefficiency. To provide negative samples, contrastive learning methods often require large batch sizes, thus regarded as sample inefficient, while noncontrastive learning methods require large representation dimensions, thus regarded as dimension inefficient. Although we have some understanding of the noncontrastive learning method, theoretical analysis of such phenomenon still remains largely unexplored. We present a theoretical analysis of the dimensional need for noncontrastive learning. We investigate the transfer between upstream representation learning and downstream tasks’ performance, demonstrating how noncontrastive methods implicitly increase interclass distances within the representation space and how the distance affects the model performance of evaluation performance. We prove that the performance of noncontrastive methods is affected by the output dimension and the number of latent classes, and illustrate why performance degrades significantly when the output dimension is substantially smaller than the number of latent classes. We demonstrate our findings through experiments on image classification experiments, and enrich the verification in audio, graph and text modalities. We also perform empirical evaluation for image models on extensive detection and segmentation tasks beyond classification that show satisfactory correspondence to our theorem.Keywords
Funding Information
- National Key Research and Development Program of China (2023YFC3306401)
- National Natural Science Foundation of China (92467108, 62141604, 62032016, 62476016)
- Beijing Nova Program (20220484106, 20230484451)
This publication has 20 references indexed in Scilit:
- What makes for uniformity for non-contrastive self-supervised learning?Science China Technological Sciences, 2022
- Multiview Deep Graph Infomax to Achieve Unsupervised Graph EmbeddingIEEE Transactions on Cybernetics, 2022
- FG-Net: A Fast and Accurate Framework for Large-Scale LiDAR Point Cloud UnderstandingIEEE Transactions on Cybernetics, 2022
- Emerging Properties in Self-Supervised Vision Transformers2021
- Contrastive Learning of General-Purpose Audio Representations2021
- Exploring Simple Siamese Representation Learning2021
- Momentum Contrast for Unsupervised Visual Representation Learning2020
- RecapNet: Action Proposal Generation Mimicking Human Cognitive ProcessIEEE Transactions on Cybernetics, 2020
- ImageNet: A large-scale hierarchical image database2009
- Universally optimal distribution of points on spheresJournal of the American Mathematical Society, 2006