Action-Stage Emphasized Spatiotemporal VLAD for Video Action Recognition
Authors:
- Zhigang TuZTZhigang Tu
- Hongyan LiHLHongyan Li
- Dejun ZhangDejun Zhang
- Justin DauwelsJDJustin Dauwels
- Baoxin LiBLBaoxin Li
- Junsong YuanJYJunsong Yuan
- 3 January 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 28 (6) , 2799-2812
Abstract
Despite outstanding performance in image recognition, convolutional neural networks (CNNs) do not yet achieve the same impressive results on action recognition in videos. This is partially due to the inability of CNN for modeling long-range temporal structures especially those involving individual action stages that are critical to human action recognition. In this paper, we propose a novel action-stage (ActionS) emphasized spatiotemporal Vector of Locally Aggregated Descriptors (ActionS-STVLAD) method to aggregate informative deep features across the entire video according to adaptive video feature segmentation and adaptive segment feature sampling (AVFS-ASFS). In our ActionSST- VLAD encoding approach, by using AVFS-ASFS, the key frame features are chosen and the corresponding deep features are automatically split into segments with the features in each segment belonging to a temporally coherent ActionS. Then, based on the extracted key frame feature in each segment, a flow-guided warping technique is introduced to detect and discard redundant feature maps, while the informative ones are aggregated by using our exploited similarity weight. Furthermore, we exploit an RGBF modality to capture motion salient regions in the RGB images corresponding to action activity. Extensive experiments are conducted on four public benchmarks - HMDB51, UCF101, Kinetics and ActivityNet for evaluation. Results show that our method is able to effectively pool useful deep features spatiotemporally, leading to state-of-the-art performance for videobased action recognition.Keywords
Funding Information
- Wuhan University (CXFW-18-413100063)
- National Key Research and Development Program of China (2016YFF0103501)
- National Natural Science Foundation of China (61572012)
- Natural Science Foundation of Hubei Province (2017CFB598, 2017CFB677)
This publication has 49 references indexed in Scilit:
- Weighted local intensity fusion method for variational optical flow estimationPattern Recognition, 2016
- ActivityNet: A large-scale video benchmark for human activity understanding2015
- Action recognition with trajectory-pooled deep-convolutional descriptors2015
- A discriminative CNN video representation for event detection2015
- A combined post-filtering method to improve accuracy of variational optical flow estimationPattern Recognition, 2014
- Action Recognition with Improved Trajectories2013
- Dense Trajectories and Motion Boundary Descriptors for Action RecognitionInternational Journal of Computer Vision, 2013
- SURF: Speeded Up Robust Features2006
- Histograms of Oriented Gradients for Human Detection2005
- Model-based temporal object verification using videoIEEE Transactions on Image Processing, 2001