Three Guidelines You Should Know for Universally Slimmable Self-Supervised Learning
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 15742-15751
Abstract
We propose universally slimmable self-supervised learning (dubbed as US3L) to achieve better accuracy-efficiency trade-offs for deploying self-supervised models across different devices. We observe that direct adaptation of self-supervised learning (SSL) to universally slimmable networks misbehaves as the training process frequently collapses. We then discover that temporal consistent guidance is the key to the success of SSL for universally slimmable networks, and we propose three guidelines for the loss design to ensure this temporal consistency from a unified gradient perspective. Moreover, we propose dynamic sampling and group regularization strategies to simultaneously improve training efficiency and accuracy. Our US3L method has been empirically validated on both convolutional neural networks and vision transformers. With only once training and one copy of weights, our method outperforms various state-of-the-art methods (individually trained or not) on benchmarks including recognition, object detection and instance segmentation.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Dynamic Slimmable Network2021
- Momentum Contrast for Unsupervised Visual Representation Learning2020
- Rethinking ImageNet Pre-Training2019
- MobileNetV2: Inverted Residuals and Linear Bottlenecks2018
- ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression2017
- Mask R-CNN2017
- Feature Pyramid Networks for Object Detection2017
- Deep Residual Learning for Image Recognition2016
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009