Saliency-context two-stream convnets for action recognition
- 19 August 2016
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3076-3080
Abstract
Recently, very deep two-stream ConvNets have achieved great discriminative power for video classification, which is especially the case for the temporal ConvNets when trained on multi-frame optical flow. However, action recognition in videos often fall prey to the wild camera motion, which poses challenges on the extraction of reliable optical flow for human body. In light of this, we propose a novel method to remove the global camera motion, which explicitly calculates a homography between two consecutive frames without human detection. Given the estimated homography due to camera motion, background motion can be canceled out from the warped optical flow. We take this a step further and design a new architecture called Saliency-Context two-stream ConvNets, where the context two-stream ConvNets are employed to recognize the entire scene in video frames, whilst the saliency streams are trained on salient human motion regions that are detected from the warped optical flow. Finally, the Saliency-Context two-stream ConvNets allow us to capture complementary information and achieve state-of-the-art performance on UCF101 dataset.Keywords
This publication has 12 references indexed in Scilit:
- Moving camera human activity localization and recognition with motionplanes and multiple homographies2015
- Caffe2014
- Large-Scale Video Classification with Convolutional Neural Networks2014
- Action Recognition with Improved Trajectories2013
- Image Classification with the Fisher Vector: Theory and PracticeInternational Journal of Computer Vision, 2013
- Better Exploiting Motion for Better Action Recognition2013
- Image Alignment and Stitching: A TutorialFoundations and Trends in Computer Graphics and Vision, 2007
- Two-Frame Motion Estimation Based on Polynomial Expansion2003
- Separate visual pathways for perception and actionTrends in Neurosciences, 1992
- Random sample consensusCommunications of the ACM, 1981