TP-VIT: A Two-Pathway Vision Transformer for Video Action Recognition
Authors:
- 23 May 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2185-2189
Abstract
Recently, inspired by the success of Transformer in natural language processing tasks, a number of works have attempted to apply Transformer-based models to video action recognition. Existing works only use one RGB stream as the input for Transformer. How to use multiple pathways and multiple streams with Transformer for action recognition has not been studied. To address this issue, we present a novel structure namely Two-Pathway Vision Transformer (TP-ViT). Two parallel spatial Transformer encoders are used as two pathways with different framerates and resolutions of the input video. The high-resolution pathway contains more spatial information, while the high-framerate pathway contains more temporal information. The two outputs are fused and fed into a temporal Transformer encoder for action recognition. Furthermore, we also fuse skeleton features into our model to get better results. Our experiments demonstrate that our proposed models achieve the state-of-the-art results on both the coarse-grained dataset Kinetics and the fine-grained dataset FineGym.Keywords
This publication has 17 references indexed in Scilit:
- VideoBERT: A Joint Model for Video and Language Representation Learning2019
- OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity FieldsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2019
- Transformer-XL: Attentive Language Models beyond a Fixed-Length Context2019
- Non-local Neural Networks2018
- The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation2018
- FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks2017
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Convolutional Two-Stream Network Fusion for Video Action Recognition2016
- FlowNet: Learning Optical Flow with Convolutional Networks2015
- ImageNet: A large-scale hierarchical image database2009