Exploring Optical-Flow-Guided Motion and Detection-Based Appearance for Temporal Sentence Grounding
- 20 January 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 25 (15209210) , 8539-8553
Abstract
Temporal sentence grounding aims to localize a target segment in an untrimmed video semantically according to a given sentence query. Most previous works focus on learning frame-level features of each whole frame 1 1In this paper, the frame is a general concept for an actual video frame or a video clip which consists of a few consecutive frames. in the entire video, and directly match them with the textual information. Such frame-level feature extraction leads to the obstacles of these methods in distinguishing ambiguous video frames with complicated contents and subtle appearance differences, thus limiting their performance. In order to differentiate fine-grained appearance similarities among consecutive frames, some state-of-the-art methods additionally employ a detection model like Faster R-CNN to obtain detailed object-level features in each frame for filtering out the redundant background contents. However, these methods suffer from missing motion analysis since the object detection module in Faster R-CNN lacks temporal modeling. To alleviate the above limitations, in this paper, we propose a novel M otion- and A ppearance-guided 3 D S emantic R easoning N etwork (MA3SRN), which incorporates optical-flow-guided motion-aware, detection-based appearance-aware, and 3D-aware object-level features to better reason the spatial-temporal object relations for accurately modelling the activity among consecutive frames. Specifically, we first develop three individual branches for motion, appearance, and 3D encoding separately to learn fine-grained motion-guided, appearance-guided, and 3D-aware object features, respectively. Then, both motion and appearance information from corresponding branches are associated to enhance the 3D-aware features for the final precise grounding. Extensive experiments on three challenging datasets (ActivityNet Caption, Charades-STA and TACoS) demonstrate that the proposed MA3SRN model achieves a new state-of-the-art.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (61972009, 61972448)
This publication has 67 references indexed in Scilit:
- Single Shot Temporal Action Detection2017
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding2016
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Deep Residual Learning for Image Recognition2016
- Natural Language Object Retrieval2016
- Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs2016
- Learning Spatiotemporal Features with 3D Convolutional Networks2015
- ActivityNet: A large-scale video benchmark for human activity understanding2015
- Glove: Global Vectors for Word Representation2014