Disentangling Inter- and Intra-Video Relations for Multi-Event Video-Text Retrieval and Grounding
- 14 November 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 34 (10577149) , 7558-7571
Abstract
Video-text retrieval aims to precisely search for videos most relevant to text queries within a video corpus. However, existing methods are largely limited to single-text (single-event) queries and are not effective at handling multi-text (multi-event) queries. Furthermore, these methods typically focus solely on retrieval and do not attempt to locate multiple events within the retrieved videos. To address these limitations, our paper proposes a novel method named Disentangling Inter- and Intra-Video Relations, which jointly addresses multi-event video-text retrieval and grounding. This method leverages both inter-video and intra-video event relationships to enhance retrieval and grounding performance. At the retrieval level, we devise a Relational Event-Centric Video-Text Retrieval module based on the principle that comprehensive textual information leads to precise correspondence between text and video. It incorporates event relationship features at different hierarchical levels and exploits the hierarchical structure of video relationships to achieve multi-level contrastive learning between events and videos. This approach enhances the richness, accuracy, and comprehensiveness of event descriptions, improving alignment precision between text and video and enabling effective differentiation among videos. For event grounding, we propose Event Contrast-Driven Video Grounding, which accounts for positional differences among events on the 2D temporal score map and achieves precise grounding of multiple events through divergence learning for their locations. Our solution not only provides efficient text-to-video retrieval but also accurately grounds events within the retrieved videos, addressing the shortcomings of existing methods. Extensive experimental results on the ActivityNet Captions and Charades-STA benchmark datasets demonstrate the superior performance of our method, validating its effectiveness. The innovation of this research lies in introducing a new joint framework for video-text retrieval and multi-event grounding while offering new ideas for further research and applications in related fields. The code is available at https://github.com/X7J92/MVT-RG.Keywords
Funding Information
- Yunnan Fundamental Research Projects (202301AV070004, 202401AS070106)
- National Natural Science Foundation of China (62276120)
This publication has 59 references indexed in Scilit:
- TS2-Net: Token Shift and Selection Transformer for Text-Video Retrieval2022
- CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioningNeurocomputing, 2022
- Anomaly detection using edge computing in video surveillance system: reviewInternational Journal of Multimedia Information Retrieval, 2022
- GHRS: Graph-based hybrid recommendation system with application to movie recommendationExpert Systems with Applications, 2022
- TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval2020
- Weakly Supervised Video Moment Retrieval From Text Queries2019
- A Joint Sequence Fusion Model for Video Question Answering and Retrieval2018
- ActivityNet: A large-scale video benchmark for human activity understanding2015
- Large-Scale Video Classification with Convolutional Neural Networks2014
- Script Data for Attribute-Based Recognition of Composite Activities2012