Coarse-to-Fine Localization of Temporal Action Proposals
- 23 September 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 22 (6) , 1577-1590
Abstract
Localizing temporal action proposals from long videos is a fundamental challenge in video analysis (e.g., action detection and recognition or dense video captioning). Most existing approaches often overlook the hierarchical granularities of actions and thus fail to discriminate fine-grained action proposals (e.g., hand washing laundry or changing a tire in vehicle repair). In this paper, we propose a novel coarse-to-fine temporal proposal (CFTP) approach to localize temporal action proposals by exploring different action granularities. Our proposed CFTP consists of three stages: a coarse proposal network (CPN) to generate long action proposals, a temporal convolutional anchor network (CAN) to localize finer proposals, and a proposal reranking network (PRN) to further identify proposals from previous stages. Specifically, CPN explores three complementary actionness curves (namely pointwise, pairwise, and recurrent curves) that represent actions at different levels for generating coarse proposals, while CAN refines these proposals by a multiscale cascaded 1D-convolutional anchor network. In contrast to existing works, our coarse-to-fine approach can progressively localize fine-grained action proposals. We conduct extensive experiments on two action benchmarks (THUMOS14 and ActivityNet v1.3) and demonstrate the superior performance of our approach when compared to the state-of-the-art techniques on various video understanding tasks.Keywords
Funding Information
- 973 Programme (2015CB351803)
This publication has 43 references indexed in Scilit:
- What Makes for Effective Detection Proposals?IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- ActivityNet: A large-scale video benchmark for human activity understanding2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Translating Videos to Natural Language Using Deep Recurrent Neural Networks2015
- Caffe2014
- Large-Scale Video Classification with Convolutional Neural Networks2014
- Action Localization with Tubelets from Motion2014
- Adaptive Learning for Celebrity Identification With Video ContextIEEE Transactions on Multimedia, 2014
- Combining the Right Features for Complex Event Recognition2013
- Temporal Localization of Actions with ActomsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013