Spatio-Temporal Adaptive Network With Bidirectional Temporal Difference for Action Recognition
- 1 March 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 33 (9) , 5174-5185
Abstract
Action Recognition is a fundamental task in computer vision field, with a wide range of applications in autonomous driving, security monitoring, etc. However, previous action recognition approaches usually suffer from the inappropriate spatio-temporal modeling or high computational consumption (e.g., 3D CNN). In this paper, we propose a novel Spatio-Temporal Adaptive Network (STANet) with bidirectional temporal difference, consisting of a Temporal Adaptive module (TA) and a Spatial Adaptive (SA) module, to sufficiently extract the crucial motion information and model the spatial pivotal appearance information from both forward and backward perspectives, respectively. Specifically, the Temporal Adaptive module uses bidirectional temporal differences to learn valuable motion trends and balance the static semantics and dynamic motion for a certain action during information fusion; while the Spatial Adaptive module uses the bidirectional temporal difference to obtain the spatio-channel attention to stress the discriminative position-relevant and semantic-relevant appearance features. Extensive experiments conducted on widely-used action recognition benchmarks UCF-101, HMDB-51, Something-Something V1, and Kinetics-400 prove the effectiveness of the proposed methods compared to other state-of-the-art approaches.Keywords
Funding Information
- National Natural Science Foundation of China (62106158, 62206010)
- State Key Laboratory of Software Development Environment (SKLSDE-2020KF-08)
- Research and Development Program of Beijing Municipal Education Commission (KM202210028007)
This publication has 53 references indexed in Scilit:
- Skeleton Optical Spectra-Based Action Recognition Using Convolutional Neural NetworksIEEE Transactions on Circuits and Systems for Video Technology, 2016
- Simple to Complex Transfer Learning for Action RecognitionIEEE Transactions on Image Processing, 2015
- Learning Spatiotemporal Features with 3D Convolutional Networks2015
- Long-term recurrent convolutional networks for visual recognition and description2015
- Action recognition with trajectory-pooled deep-convolutional descriptors2015
- Going deeper with convolutions2015
- Large-Scale Video Classification with Convolutional Neural Networks2014
- Action Recognition with Improved Trajectories2013
- 3D Convolutional Neural Networks for Human Action RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2012
- Long Short-Term MemoryNeural Computation, 1997