SVFormer: Semi-supervised Video Transformer for Action Recognition
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 18816-18826
Abstract
Semi-supervised action recognition is a challenging but critical task due to the high cost of video annotations. Existing approaches mainly use convolutional neural networks, yet current revolutionary vision transformer models have been less explored. In this paper, we investigate the use of transformer models under the SSL setting for action recognition. To this end, we introduce SVFormer, which adopts a steady pseudo-labeling framework (i.e., EMA-Teacher) to cope with unlabeled video samples. While a wide range of data augmentations have been shown effective for semi-supervised image classification, they generally produce limited results for video recognition. We therefore introduce a novel augmentation strategy, Tube Token-Mix, tailored for video data where video clips are mixed via a mask with consistent masked tokens over the temporal axis. In addition, we propose a temporal warping augmentation to cover the complex temporal variation in videos, which stretches selected frames to various temporal durations in the clip. Extensive experiments on three datasets Kinetics-400, UCF-101, and HMDB-51 verify the advantage of SVFormer. In particular, SVFormer outperforms the state-of-the-art by 31.5% with fewer training epochs under the 1% labeling rate of Kinetics-400. Our method can hopefully serve as a strong benchmark and encourage future search on semi-supervised action recognition with Transformer networks. Code is released at https://github.com/ChenHsing/SVFormer.Keywords
All Related Versions
Funding Information
- NSFC (62032006,62102092)
This publication has 43 references indexed in Scilit:
- Charting the Right Manifold: Manifold Mixup for Few-shot Learning2020
- CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features2019
- SlowFast Networks for Video Recognition2019
- Interpolation Consistency Training for Semi-supervised Learning2019
- AutoAugment: Learning Augmentation Strategies From Data2019
- Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition2017
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- HMDB: A large video database for human motion recognition2011
- ImageNet: A large-scale hierarchical image database2009
- The computation of optical flowACM Computing Surveys, 1995