Semi-Supervised Masked Autoencoders: Unlocking Vision Transformer Potential with Limited Data
Authors:
- Atik FaysalAFAtik Faysal
- Mohammad RostamiMRMohammad Rostami
- Reihaneh Gh. RoshanRRReihaneh Gh. Roshan
- Nikhil MuralidharNMNikhil Muralidhar
- Huaxia WangHWHuaxia Wang
Preprint
Abstract
We address the challenge of training Vision Transformers (ViTs) when labeled data is scarce but unlabeled data is abundant. We propose Semi-Supervised Masked Autoencoder (SSMAE), a framework that jointly optimizes masked image reconstruction and classification using both unlabeled and labeled samples with dynamically selected pseudo-labels. SSMAE introduces a validation-driven gating mechanism that activates pseudo-labeling only after the model achieves reliable, high-confidence predictions that are consistent across both weakly and strongly augmented views of the same image, reducing confirmation bias. On CIFAR-10 and CIFAR-100, SSMAE consistently outperforms supervised ViT and fine-tuned MAE, with the largest gains in low-label regimes (+9.24% over ViT on CIFAR-10 with 10% labels). Our results demonstrate that when pseudo-labels are introduced is as important as how they are generated for data-efficient transformer training. Codes are available at https://github.com/atik666/ssmae.Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- Co-training with High-Confidence Pseudo Labels for Semi-supervised Medical Image Segmentation2023
- SCMT: Self-Correction Mean Teacher for Semi-supervised Object Detection2022
- SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders2022
- Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic SegmentationInternational Journal of Computer Vision, 2021
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019