VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking
Authors:
Top Cited Papers
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 14549-14560
Abstract
Scale is the primary factor for building a powerful foundation model that could well generalize to a variety of downstream tasks. However, it is still challenging to train video foundation models with billions of parameters. This paper shows that video masked autoencoder (VideoMAE) is a scalable and general self-supervised pre-trainer for building video foundation models. We scale the VideoMAE in both model and data with a core design. Specifically, we present a dual masking strategy for efficient pre-training, with an encoder operating on a subset of video tokens and a decoder processing another subset of video tokens. Although VideoMAE is very efficient due to high masking ratio in encoder, masking decoder can still further reduce the overall computational cost. This enables the efficient pre-training of billion-level models in video. We also use a progressive training paradigm that involves an initial pre-training on a diverse multi-sourced unlabeled dataset, followed by a post-pre-training on a mixed labeled dataset. Finally, we successfully train a video ViT model with a billion parameters, which achieves a new state-of-the-art performance on the datasets of Kinetics (90.0% on K400 and 89.9% on K600) and Something-Something (68.7% on V1 and 77.0% on V2). In addition, we extensively verify the pre-trained video ViT models on a variety of downstream tasks, demonstrating its effectiveness as a general video representation learner.Keywords
All Related Versions
Funding Information
- National Key R&D Program of China (2022ZD0160900,2022ZD0160100)
- National Natural Science Foundation of China (62076119,61921006)
- Youth Innovation Promotion Association of Chinese Academy of Sciences (2020355)
This publication has 50 references indexed in Scilit:
- The “Something Something” Video Database for Learning and Evaluating Visual Common Sense2017
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- The THUMOS challenge on action recognition for videos “in the wild”Computer Vision and Image Understanding, 2017
- Context Encoders: Feature Learning by Inpainting2016
- Deep Residual Learning for Image Recognition2016
- Going deeper with convolutions2015
- Extracting and composing robust features with denoising autoencoders2008
- Histograms of Oriented Gradients for Human Detection2005
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998