Dual-Task Mutual Reinforcing Embedded Joint Video Paragraph Retrieval and Grounding
- 30 April 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 5879-5894
Abstract
Video Paragraph Grounding (VPG) aims to precisely locate the most appropriate moments within a video that are relevant to a given textual paragraph query. However, existing methods typically rely on large-scale annotated temporal labels and assume that the correspondence between videos and paragraphs is known. This is impractical in real-world applications, as constructing temporal labels requires significant labor costs, and the correspondence is often unknown. To address this issue, we propose a Dual-task Mutual Reinforcing Embedded Joint Video Paragraph Retrieval and Grounding method (DMR-JRG). In this method, retrieval and grounding tasks are mutually reinforced rather than being treated as separate issues. DMR-JRG mainly consists of two branches: a retrieval branch and a grounding branch. The retrieval branch uses inter-video contrastive learning to roughly align the global features of paragraphs and videos, reducing modality differences and constructing a coarse-grained feature space to break free from the need for correspondence between paragraphs and videos. Additionally, this coarse-grained feature space further facilitates the grounding branch in extracting fine-grained contextual representations. In the grounding branch, we achieve precise cross-modal matching and grounding by exploring the consistency between local, global, and temporal dimensions of video segments and textual paragraphs. By synergizing these dimensions, we construct a fine-grained feature space for video and textual features, greatly reducing the need for large-scale annotated temporal labels. Meanwhile, we design a grounding reinforcement retrieval module (GRRM) that brings the coarse-grained feature space of the retrieval branch closer to the fine-grained feature space of the grounding branch, thereby reinforcing retrieval branch through grounding branch, and finally achieving mutual reinforcement between tasks. Extensive experiments on three challenging datasets demonstrate the effectiveness of our proposed method. The code is available at https://github.com/X7J92/DMR-JRG.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62276120, 62272133)
- Yunnan Fundamental Research Projects (202301AV070004, 202401AS070106)
- Shenzhen Science and Technology Program (KJZD20230923114600002)
- Shenzhen Colleges and Universities Stable Support Program (GXWD20220811170100001)
- National Natural Science Foundation of China (62276120, 62272133)
- Yunnan Fundamental Research Projects (202301AV070004, 202401AS070106)
- Shenzhen Science and Technology Program (KJZD20230923114600002)
- Shenzhen Colleges and Universities Stable Support Program (GXWD20220811170100001)
This publication has 67 references indexed in Scilit:
- End-to-end dense video grounding via parallel regressionComputer Vision and Image Understanding, 2024
- Mutual prediction learning and mixed viewpoints for unsupervised-domain adaptation person re-identification on blockchainSimulation Modelling Practice and Theory, 2022
- YOLOv3-MT: A YOLOv3 using multi-target tracking for vehicle visual detectionApplied Intelligence, 2021
- TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval2020
- End-to-End Object Detection with Transformers2020
- DeepMTT: A deep learning maneuvering target-tracking algorithm based on bidirectional LSTM networkInformation Fusion, 2020
- FaceNet: A unified embedding for face recognition and clustering2015
- ActivityNet: A large-scale video benchmark for human activity understanding2015
- Large-Scale Video Classification with Convolutional Neural Networks2014
- Script Data for Attribute-Based Recognition of Composite Activities2012