E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding
Authors:
Preprint
Abstract
Recent advances in Video Large Language Models (Video-LLMs) have demonstrated their great potential in general-purpose video understanding. To verify the significance of these models, a number of benchmarks have been proposed to diagnose their capabilities in different scenarios. However, existing benchmarks merely evaluate models through video-level question-answering, lacking fine-grained event-level assessment and task diversity. To fill this gap, we introduce E.T. Bench (Event-Level & Time-Sensitive Video Understanding Benchmark), a large-scale and high-quality benchmark for open-ended event-level video understanding. Categorized within a 3-level task taxonomy, E.T. Bench encompasses 7.3K samples under 12 tasks with 7K videos (251.4h total length) under 8 domains, providing comprehensive evaluations. We extensively evaluated 8 Image-LLMs and 12 Video-LLMs on our benchmark, and the results reveal that state-of-the-art models for coarse-level (video-level) understanding struggle to solve our fine-grained tasks, e.g., grounding event-of-interests within videos, largely due to the short video context length, improper time representations, and lack of multi-event training data. Focusing on these issues, we further propose a strong baseline model, E.T. Chat, together with an instruction-tuning dataset E.T. Instruct 164K tailored for fine-grained event-level understanding. Our simple but effective solution demonstrates superior performance in multiple scenarios.Keywords
All Related Versions
This publication has 35 references indexed in Scilit:
- VideoChat: chat-centric video understandingScience China Information Sciences, 2025
- AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answering2024
- $\mathrm R^2$-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Grounding2024
- HowToCaption: Prompting LLMs to Transform Video Annotations at Scale2024
- Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models2024
- LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models2024
- ActivityNet: A large-scale video benchmark for human activity understanding2015
- TVSum: Summarizing web videos using titles2015
- Ranking Domain-Specific Highlights by Analyzing Edited Videos2014
- Creating Summaries from User Videos2014