Token Boosting for Robust Self-Supervised Visual Transformer Pre-training
Preprint
Abstract
Learning with large-scale unlabeled data has become a powerful tool for pre-training Visual Transformers (VTs). However, prior works tend to overlook that, in real-world scenarios, the input data may be corrupted and unreliable. Pre-training VTs on such corrupted data can be challenging, especially when we pre-train via the masked autoencoding approach, where both the inputs and masked ``ground truth" targets can potentially be unreliable in this case. To address this limitation, we introduce the Token Boosting Module (TBM) as a plug-and-play component for VTs that effectively allows the VT to learn to extract clean and robust features during masked autoencoding pre-training. We provide theoretical analysis to show how TBM improves model pre-training with more robust and generalizable representations, thus benefiting downstream tasks. We conduct extensive experiments to analyze TBM's effectiveness, and results on four corrupted datasets demonstrate that TBM consistently improves performance on downstream tasks.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Image as a Foreign Language: BEIT Pretraining for Vision and Vision-Language Tasks2023
- Masked Discrimination for Self-supervised Learning on Point Clouds2022
- MeshMAE: Masked Autoencoders for 3D Mesh Data Analysis2022
- P-STMO: Pre-trained Spatial Temporal Many-to-One Model for 3D Human Pose Estimation2022
- Hierarchically Self-supervised Transformer for Human Skeleton Representation Learning2022
- BEVT: BERT Pretraining of Video Transformers2022
- Masked Autoencoders Are Scalable Vision Learners2022
- Private and common feature learning with adversarial network for RGBD object classificationNeurocomputing, 2021
- Object Detection Under Rainy Conditions for Autonomous Vehicles: A Review of State-of-the-Art and Emerging TechniquesIEEE Signal Processing Magazine, 2020
- View Transfer on Human Skeleton Pose: Automatically Disentangle the View-Variant and View-Invariant Information for Pose Representation LearningInternational Journal of Computer Vision, 2020