The Role of Entropy and Reconstruction in Multi-View Self-Supervised Learning
Authors:
- Borja Rodríguez-GálvezBRBorja Rodríguez-Gálvez
- Arno BlaasABArno Blaas
- Pau RodríguezPRPau Rodríguez
- Adam GolińskiAGAdam Goliński
- Xavier SuauXSXavier Suau
- Jason RamapuramJRJason Ramapuram
- Dan BusbridgeDBDan Busbridge
- Luca ZappellaLZLuca Zappella
Preprint
Abstract
The mechanisms behind the success of multi-view self-supervised learning (MVSSL) are not yet fully understood. Contrastive MVSSL methods have been studied through the lens of InfoNCE, a lower bound of the Mutual Information (MI). However, the relation between other MVSSL methods and MI remains unclear. We consider a different lower bound on the MI consisting of an entropy and a reconstruction term (ER), and analyze the main MVSSL families through its lens. Through this ER bound, we show that clustering-based methods such as DeepCluster and SwAV maximize the MI. We also re-interpret the mechanisms of distillation-based approaches such as BYOL and DINO, showing that they explicitly maximize the reconstruction term and implicitly encourage a stable entropy, and we confirm this empirically. We show that replacing the objectives of common MVSSL methods with this ER bound achieves competitive performance, while making them stable when training with smaller batch sizes or smaller exponential moving average (EMA) coefficients. Github repo: https://github.com/apple/ml-entropy-reconstruction.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- The Information Bottleneck Problem and its Applications in Machine LearningIEEE Journal on Selected Areas in Information Theory, 2020
- The Convex Information Bottleneck LagrangianEntropy, 2020
- Convergence properties of functional estimates for discrete distributionsRandom Structures & Algorithms, 2001
- Estimation of entropy and other functionals of a multivariate densityAnnals of the Institute of Statistical Mathematics, 1989
- Information Theory and Reliable Communication1972
- On a Property of Non-Degenerate Diffusion ProcessesTheory of Probability & Its Applications, 1959