SELongVLM: Empowering Long Video Language Models With Self-Corrective Clip Selection
- 1 July 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (7) , 8694-8709
Abstract
Recent advances in multimodal large language models (MLLMs) have enabled impressive progress in visual-language reasoning, yet long-video understanding remains a formidable challenge due to the need for coherent reasoning over ultra-long spatiotemporal dependencies. Existing methods struggle with the vast candidate space for relevant information in long videos, often failing to distinguish meaningful events from redundant content. We identify two critical and previously under-explored issues: absolute redundancy, where static visual content inflates token counts without adding narrative value, and relative redundancy, where task-irrelevant segments introduce noise that impairs reasoning. Compounding these issues is the weak spatiotemporal modeling in current MLLMs, which limits their ability to capture complex event dynamics. To address these multifaceted challenges, we introduce SELongVLM, a dynamically lenient-to-stringent selection long video language model. SELongVLM integrates two coordinated branches: a Residual Token Pruner (RTP) that removes repetitive background tokens via inter-frame residual modeling thus mitigating absolute redundancy while preserving motion cues, and a Semantic-aware Self-Correction Selector (SCSelector) that progressively refines query-relevant clip selection without frame-level annotations to reduce relative redundancy, guided by a stringent-to-lenient self-correcting mechanism during optimization. To ensure causal continuity and bolster spatiotemporal reasoning across disjoint clips, the framework further incorporates an action-aware operation for intra-clip dynamics and a temporal memory for cross-clip context, enabling robust spatiotemporal inference on long videos. Extensive experiments across eight benchmarks demonstrate that SELongVLM markedly outperforms existing models on both general and specialized long-video tasks. Specifically, it achieves 65.5% on VideoMME and 69.8% on MLVU for general benchmarks, and delivers strong performance on four specialized benchmarks - for example, 39.2% on TOMATO for fine-grained temporal reasoning and 69.2% on EventBench for event-level understanding.Keywords
Funding Information
- New Generation Artificial Intelligence-National Science and Technology Major Projection (2025ZD0123100)
- National Natural Science Foundation of China (62573399)
- National Natural Science Foundation of China (U25A20530)
This publication has 36 references indexed in Scilit:
- InternVideo2: Scaling Foundation Models for Multimodal Video Understanding2024
- VideoAgent: Long-Form Video Understanding with Large Language Model as Agent2024
- LongVLM: Efficient Long Video Understanding via Large Language Models2024
- LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models2024
- ST-LLM: Large Language Models Are Effective Temporal Learners2024
- MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding2024
- VTimeLLM: Empower LLM to Grasp Video Moments2024
- Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding2024
- NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions2021
- Study on density peaks clustering based on k-nearest neighbors and principal component analysisKnowledge-Based Systems, 2016