Semi-Supervised Vision Transformers
Preprint
Abstract
We study the training of Vision Transformers for semi-supervised image classification. Transformers have recently demonstrated impressive performance on a multitude of supervised learning tasks. Surprisingly, we show Vision Transformers perform significantly worse than Convolutional Neural Networks when only a small set of labeled data is available. Inspired by this observation, we introduce a joint semi-supervised learning framework, Semiformer, which contains a transformer stream, a convolutional stream and a carefully designed fusion module for knowledge sharing between these streams. The convolutional stream is trained on limited labeled data and further used to generate pseudo labels to supervise the training of the transformer stream on unlabeled data. Extensive experiments on ImageNet demonstrate that Semiformer achieves 75.5% top-1 accuracy, outperforming the state-of-the-art by a clear margin. In addition, we show, among other things, Semiformer is a general framework that is compatible with most modern transformer and convolutional neural architectures. Code is available at https://github.com/wengzejia1/Semiformer.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Efficient Video Transformers with Spatial-Temporal Token Selection2022
- ConViT: improving vision transformers with soft convolutional inductive biases*Journal of Statistical Mechanics: Theory and Experiment, 2022
- BEVT: BERT Pretraining of Video Transformers2022
- CvT: Introducing Convolutions to Vision Transformers2021
- Meta Pseudo Labels2021
- Self-Training With Noisy Student Improves ImageNet Classification2020
- S4L: Self-Supervised Semi-Supervised Learning2019
- Non-local Neural Networks2018
- Deep Residual Learning for Image Recognition2016
- Semi-Supervised Self-Training of Object Detection Models2005