VTimeLLM: Empower LLM to Grasp Video Moments
Authors:
Top Cited Papers
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 14271-14280
Abstract
Large language models (LLMs) have shown remarkable text understanding capabilities, which have been ex-tended as Video LLMs to handle video data for compre-hending visual details. However, existing Video LLMs can only provide a coarse description of the entire video, failing to capture the precise start and end time bound-ary of specific events. In this paper, we solve this issue via proposing VTimeLLM, a novel Video LLM designed for fine-grained video moment understanding and reasoning with respect to time boundary. Specifically, our VTimeLLM adopts a boundary-aware three-stage training strategy, which respectively utilizes image-text pairs for feature alignment, multiple-event videos to increase temporal-boundary awareness, and high-quality video-instruction tuning to further improve temporal understanding ability as well as align with human intents. Extensive experiments demonstrate that in fine-grained time-related comprehension tasks for videos such as Temporal Video Grounding and Dense Video Captioning, VTimeLLM significantly outperforms existing Video LLMs. Besides, benefits from the fine-grained temporal understanding of the videos further enable VTimeLLM to beat existing Video LLMs in video di-alogue benchmark, showing its superior cross-modal understanding and reasoning abilities. 1 1 Our project page is at https://github.com/huangb23/VTimeLLMKeywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning2023
- Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval2021
- SODA: Story Oriented Dense Video Captioning Evaluation Framework2020
- MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment2019
- Bidirectional Attentive Fusion with Context Gating for Dense Video Captioning2018
- End-to-End Dense Video Captioning with Masked Transformer2018
- Localizing Moments in Video with Natural Language2017
- TALL: Temporal Activity Localization via Language Query2017
- Dense-Captioning Events in Videos2017
- CIDEr: Consensus-based image description evaluation2015