Active Temporal Action Detection in Untrimmed Videos Via Deep Reinforcement Learning
Authors:
Open Access
- 28 September 2018
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 6, 59126-59140
Abstract
Existing action detection algorithms usually generate action proposals through an extensive search over the video at multiple temporal scales, which brings about huge computational overhead and deviates from the human perception procedure. We argue that the process of detecting actions in the video should be naturally one of observation and refinement: observe the current temporal window and refine the span of attended window to cover true action regions. In this paper, we propose an active action detection model that learns to search actions through continuously adjusting the bounds of temporal attended window in a self-adaptive way. The whole process can be deemed as an exploring procedure, where an agent is first placed at the beginning of the video and then traverses the whole video by adopting a sequence of transformations on the current attended window to discover actions according to a learned policy. We utilize reinforcement learning, especially the deep Q-learning algorithm to learn the agent’s decision policy. Actually, we construct an end-to-end trainable framework for the action detection task, which includes a proposal generation network based on deep Q-learning, and the classification and regression networks responsible for the action category prediction and the action location adjustment, respectively. In addition, we design a long short-term memory structure upon extracted convolutional neural network features of sparsely sampled frames to generate the effective feature representations for video sequences of various durations. We evaluate the action proposal performance of our approach on THUMOS’14 and assess the generalization ability for unseen action categories on ActivityNet . We also compare the action detection performance of ours with other state-of-the-art methods on both data sets. Experiment results validate the effectiveness of the proposed approach, which can achieve comparative or superior performance than other action detection methods via much fewer proposals.Keywords
Funding Information
- National Natural Science Foundation of China (61602014)
- Shenzhen Municipal Fundamental Research Program (JCYJ20170818141120312)
- National Natural Science Foundation of China
- Guangdong Province Scientific Research on Big Data (U1611461)
This publication has 38 references indexed in Scilit:
- Bag-of-Fragments2015
- Human-level control through deep reinforcement learningNature, 2015
- Learning Spatial and Temporal Extents of Human Actions for Action DetectionIEEE Transactions on Multimedia, 2015
- Poselet Key-Framing: A Model for Human Activity Recognition2013
- Selective Search for Object RecognitionInternational Journal of Computer Vision, 2013
- Dense Trajectories and Motion Boundary Descriptors for Action RecognitionInternational Journal of Computer Vision, 2013
- Human activity analysisACM Computing Surveys, 2011
- A survey of vision-based methods for action representation, segmentation and recognitionComputer Vision and Image Understanding, 2011
- Distinctive Image Features from Scale-Invariant KeypointsInternational Journal of Computer Vision, 2004
- Long Short-Term MemoryNeural Computation, 1997