VisionZip: Longer is Better but Not Necessary in Vision Language Models
Authors:
Top Cited Papers
- 10 June 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 19792-19802
Abstract
Recent advancements in vision-language models have enhanced performance by increasing the length of visual tokens, making them much longer than text tokens and significantly raising computational costs. However, we observe that the visual tokens generated by popular vision encoders, such as CLIP and SigLIP, contain significant redundancy. To address this, we introduce VisionZip, a simple yet effective method that selects a set of informative tokens for input to the language model, reducing visual token redundancy and improving efficiency while maintaining model performance. The proposed VisionZip can be widely applied to image and video understanding tasks and is well-suited for multi-turn dialogues in real-world scenarios, where previous methods tend to underperform. Experimental results show that VisionZip outperforms the previous state-of-the-art method by at least 5% performance gains across nearly all settings. Moreover, our method significantly enhances model inference speed, improving the prefilling time by 8× and enabling the LLaVA-Next 13B model to infer faster than the LLaVA-Next 7B model while achieving better results. Furthermore, we analyze the causes of this redundancy and encourage the community to focus on extracting better visual features rather than merely increasing token length. Our code is available at https://github.com/dvlab-research/VisionZip.Keywords
All Related Versions
Funding Information
- Shenzhen Science and Technology Innovation Program
This publication has 32 references indexed in Scilit:
- Mobile Edge Intelligence for Large Language Models: A Contemporary SurveyIEEE Communications Surveys & Tutorials, 2025
- ShareGPT4V: Improving Large Multi-modal Models with Better Captions2024
- An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models2024
- LITA: Language Instructed Temporal-Localization Assistant2024
- Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentation2024
- LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models2024
- VLP: A Survey on Vision-language Pre-trainingMachine Intelligence Research, 2023
- VizWiz Grand Challenge: Answering Visual Questions from Blind People2018
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering2017
- TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering2017