Spatiotemporal Pyramid Network for Video Action Recognition
Authors:
- Yunbo WangYWYunbo Wang
- Mingsheng LongMLMingsheng Long
- Jianmin WangJWJianmin Wang
- Philip S. YuPYPhilip S. Yu
- 1 July 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 10636919,p. 2097-2106
Abstract
Two-stream convolutional networks have shown strong performance in video action recognition tasks. The key idea is to learn spatiotemporal features by fusing convolutional networks spatially and temporally. However, it remains unclear how to model the correlations between the spatial and temporal structures at multiple abstraction levels. First, the spatial stream tends to fail if two videos share similar backgrounds. Second, the temporal stream may be fooled if two actions resemble in short snippets, though appear to be distinct in the long term. We propose a novel spatiotemporal pyramid network to fuse the spatial and temporal features in a pyramid structure such that they can reinforce each other. From the architecture perspective, our network constitutes hierarchical fusion strategies which can be trained as a whole using a unified spatiotemporal loss. A series of ablation experiments support the importance of each fusion strategy. From the technical perspective, we introduce the spatiotemporal compact bilinear operator into video analysis tasks. This operator enables efficient training of bilinear fusion operations which can capture full interactions between the spatial and temporal features. Our final network achieves state-of-the-art results on standard video datasets.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Real-Time Action Recognition with Enhanced Motion Vector CNNs2016
- Actions ~ Transformations2016
- Learning to Track for Spatio-Temporal Action Localization2015
- Beyond short snippets: Deep networks for video classification2015
- Action recognition with trajectory-pooled deep-convolutional descriptors2015
- Caffe2014
- Large-Scale Video Classification with Convolutional Neural Networks2014
- Action Recognition with Improved Trajectories2013
- 3D Convolutional Neural Networks for Human Action RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2012
- HMDB: A large video database for human motion recognition2011