Unmasked Teacher: Towards Training-Efficient Video Foundation Models
Authors:
Top Cited Papers
- 1 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 19891-19903
Abstract
Video Foundation Models (VFMs) have received limited exploration due to high computational costs and data scarcity. Previous VFMs rely on Image Foundation Models (IFMs), which face challenges in transferring to the video domain. Although VideoMAE has trained a robust ViT from limited data, its low-level reconstruction poses convergence difficulties and conflicts with high-level cross-modal alignment. This paper proposes a training-efficient method for temporal-sensitive VFMs that integrates the benefits of existing methods. To increase data efficiency, we mask out most of the low-semantics video tokens, but selectively align the unmasked tokens with IFM, which serves as the UnMasked Teacher (UMT). By providing semantic guidance, our method enables faster convergence and multi-modal friendliness. With a progressive pre-training framework, our model can handle various tasks including scene-related, temporal-related, and complex video-language understanding. Using only public sources for pre-training in 6 days on 32 A100 GPUs, our scratch-built ViT-L/16 achieves state-of-the-art performances on various video tasks.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China
- National Natural Science Foundation of China
- Chinese Academy of Sciences
This publication has 45 references indexed in Scilit:
- Context Autoencoder for Self-supervised Representation LearningInternational Journal of Computer Vision, 2023
- MVP: Multimodality-Guided Visual Pre-training2022
- Expanding Language-Image Pretrained Models for General Video Recognition2022
- CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioningNeurocomputing, 2022
- Semantic Understanding of Scenes Through the ADE20K DatasetInternational Journal of Computer Vision, 2018
- A Joint Sequence Fusion Model for Video Question Answering and Retrieval2018
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- The THUMOS challenge on action recognition for videos “in the wild”Computer Vision and Image Understanding, 2017
- Movie DescriptionInternational Journal of Computer Vision, 2017
- Temporal Segment Networks: Towards Good Practices for Deep Action Recognition2016