Semantic Cues Enhanced Multimodality Multistream CNN for Action Recognition
- 25 April 2018
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 29 (5) , 1423-1437
Abstract
This paper addresses the issue of video-based action recognition by exploiting an advanced multi-stream Convolutional Neural Network (CNN) to fully use semantics-derived multiple modalities in both spatial (appearance) and temporal (motion) domains, since the performance of the CNN-based action recognition methods heavily relate to two factors: semantic visual cues and the network architecture. Our work consists of two major parts. First, to extract useful human-related semantics accurately, we propose a novel spatiotemporal saliency based video object segmentation (STS-VOS) model. By fusing different distinctive saliency maps, which are computed according to object signatures of complementary object detection approaches, a refined spatiotemporal saliency maps can be obtained. In this way, various challenges in the realistic video can be handled jointly. Based on the estimated saliency maps, an energy function is constructed to segment two semantic cues: the actor and one distinctive acting part of the actor. Second, we modify the architecture of the two-stream network (TS-Net) to design a multi-stream network (MS-Net) that consists of three TS-Nets with respect to the extracted semantics, which is able to use deeper abstract visual features of multi-modalities in multi-scale spatiotemporally. Importantly, the performance of action recognition is significantly boosted when integrating the captured human-related semantics into our framework. Experiments on four public benchmarks JHMDB, HMDB51, UCF-Sports and UCF101 demonstrate that the proposed method outperforms the state of the art algorithms.Keywords
Funding Information
- Singapore Ministry of Education Academic Research Fund Tier 2 (MOE2015-T2-2-114)
- National Natural Science Foundation of China (61501198)
- Natural Science Foundation of Hubei Province (2014CFB461)
- University at Buffalo
This publication has 54 references indexed in Scilit:
- Evaluating Two-Stream CNN for Video Classification2015
- Discovering Primary Objects in Videos by Saliency Fusion and Iterative Appearance EstimationIEEE Transactions on Circuits and Systems for Video Technology, 2015
- A combined post-filtering method to improve accuracy of variational optical flow estimationPattern Recognition, 2014
- Block-Sparse RPCA for Salient Motion DetectionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2014
- Return of the Devil in the Details: Delving Deep into Convolutional Nets2014
- Selective Search for Object RecognitionInternational Journal of Computer Vision, 2013
- Motion Coherent Tracking Using Multi-label MRF OptimizationInternational Journal of Computer Vision, 2011
- Visual attention detection in video sequences using spatiotemporal cues2006
- Detecting salient motion by accumulating directionally-consistent flowIEEE Transactions on Pattern Analysis and Machine Intelligence, 2000
- Long Short-Term MemoryNeural Computation, 1997