Context-Enhanced Video Moment Retrieval with Large Language Models
Preprint
Abstract
Current methods for Video Moment Retrieval (VMR) struggle to align complex situations involving specific environmental details, character descriptions, and action narratives. To tackle this issue, we propose a Large Language Model-guided Moment Retrieval (LMR) approach that employs the extensive knowledge of Large Language Models (LLMs) to improve video context representation as well as cross-modal alignment, facilitating accurate localization of target moments. Specifically, LMR introduces a context enhancement technique with LLMs to generate crucial target-related context semantics. These semantics are integrated with visual features for producing discriminative video representations. Finally, a language-conditioned transformer is designed to decode free-form language queries, on the fly, using aligned video representations for moment retrieval. Extensive experiments demonstrate that LMR achieves state-of-the-art results, outperforming the nearest competitor by up to 3.28\% and 4.06\% on the challenging QVHighlights and Charades-STA benchmarks, respectively. More importantly, the performance gains are significantly higher for localization of complex queries.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- VideoChat: chat-centric video understandingScience China Information Sciences, 2025
- Toward Complex-query Referring Image Segmentation: A Novel BenchmarkACM Transactions on Multimedia Computing, Communications, and Applications, 2024
- Contextual Object Detection with Multimodal Large Language ModelsInternational Journal of Computer Vision, 2024
- Improved Baselines with Visual Instruction Tuning2024
- LISA: Reasoning Segmentation via Large Language Model2024
- BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual QuestionsProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models2023
- Query-aware video encoder for video moment retrievalNeurocomputing, 2022
- Span-based Localizing Network for Natural Language Video Localization2020
- TALL: Temporal Activity Localization via Language Query2017