On the Importance of Asymmetry for Siamese Representation Learning
Authors:
- Xiao WangXWXiao Wang
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 16549-16558
Abstract
Many recent self-supervised frameworks for visual representation learning are based on certain forms of Siamese networks. Such networks are conceptually symmetric with two parallel encoders, but often practically asymmetric as numerous mechanisms are devised to break the symmetry. In this work, we conduct a formal study on the importance of asymmetry by explicitly distinguishing the two encoders within the network - one produces source encodings and the other targets. Our key insight is keeping a relatively lower variance in target than source generally benefits learning. This is empirically justified by our results from five case studies covering different variance-oriented designs, and is aligned with our preliminary theoretical analysis on the baseline. Moreover, we find the improvements from asymmetric designs generalize well to longer training schedules, multiple other frameworks and newer backbones. Finally, the combined effect of several asymmetric designs achieves a state-of-the-art accuracy on ImageNet linear probing and competitive results on downstream transfer. We hope our exploration will inspire more research in exploiting asymmetry for Siamese representation learning.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Emerging Properties in Self-Supervised Vision Transformers2021
- Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning2021
- Exploring Simple Siamese Representation Learning2021
- Momentum Contrast for Unsupervised Visual Representation Learning2020
- Self-Supervised Learning of Pretext-Invariant Representations2020
- Randaugment: Practical automated data augmentation with a reduced search space2020
- Iterative Learning with Open-set Noisy Labels2018
- Deep Residual Learning for Image Recognition2016
- DeepFace: Closing the Gap to Human-Level Performance in Face Verification2014
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009