VTimeLLM: Empower LLM to Grasp Video Moments
Preprint
Abstract
Large language models (LLMs) have shown remarkable text understanding capabilities, which have been extended as Video LLMs to handle video data for comprehending visual details. However, existing Video LLMs can only provide a coarse description of the entire video, failing to capture the precise start and end time boundary of specific events. In this paper, we solve this issue via proposing VTimeLLM, a novel Video LLM designed for fine-grained video moment understanding and reasoning with respect to time boundary. Specifically, our VTimeLLM adopts a boundary-aware three-stage training strategy, which respectively utilizes image-text pairs for feature alignment, multiple-event videos to increase temporal-boundary awareness, and high-quality video-instruction tuning to further improve temporal understanding ability as well as align with human intents. Extensive experiments demonstrate that in fine-grained time-related comprehension tasks for videos such as Temporal Video Grounding and Dense Video Captioning, VTimeLLM significantly outperforms existing Video LLMs. Besides, benefits from the fine-grained temporal understanding of the videos further enable VTimeLLM to beat existing Video LLMs in video dialogue benchmark, showing its superior cross-modal understanding and reasoning abilities.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- VideoChat: chat-centric video understandingScience China Information Sciences, 2025
- Improved Baselines with Visual Instruction Tuning2024
- Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models2024
- A Closer Look at Debiased Temporal Sentence Grounding in Videos: Dataset, Metric, and ApproachACM Transactions on Multimedia Computing, Communications, and Applications, 2023
- Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding2023
- A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformer2020
- Span-based Localizing Network for Natural Language Video Localization2020
- ExCL: Extractive Clip Localization Using Natural Language Descriptions2019