Spatial-Temporal Action Localization With Hierarchical Self-Attention
- 5 February 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 24 (15209210) , 625-639
Abstract
This paper proposes a novel architecture for spatial-temporal action localization in videos. The new architecture first employs a two-stream 3D convolutional neural network (3D-CNN) to provide initial action detection. Next, a new Hierarchical Self-Attention Network (HiSAN), the core of this architecture, is developed to learn the spatial-temporal relationships of key actors. Spatial Gaussian priors (SGP) are also imbued to the bidirectional self-attention to enhance HiSAN in modelling the relationships of neighboring actors. Such a combination of 3D-CNN and SGP augmented HiSAN allows us to effectively extract both of the spatial context information and the long-term temporal dependency to improve action localization accuracy. Afterwards, a new fusion strategy is employed, which first re-scores the bounding boxes to settle the inconsistent detection scores caused by background clutter or occlusion, and then aggregates the motion and appearance information from the two-stream network with the motion saliency to alleviate the impact of camera movement. Finally, a tube association network based on the self-similarity of the actors’ appearance and spatial information across frames is addressed to efficaciously construct the action tubes. Simulations on four widespread datasets reveal the efficacy of the new approach.Keywords
Funding Information
- Ministry of Science and Technology (MOST 109-2221-E-011-116, MOST 109-2221-E-011-131)
This publication has 60 references indexed in Scilit:
- Multi-region Two-Stream R-CNN for Action Detection2016
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Actionness Estimation Using Hybrid Fully Convolutional Networks2016
- Learning to Track for Spatio-Temporal Action Localization2015
- Learning to Track: Online Multi-object Tracking by Decision Making2015
- Finding action tubes2015
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- Towards Understanding Action Recognition2013
- Intelligent visual surveillance — A surveyInternational Journal of Control, Automation, and Systems, 2010
- High Accuracy Optical Flow Estimation Based on a Theory for Warping2004