VisionZip: Longer is Better but Not Necessary in Vision Language Models
Authors:
Preprint
Abstract
Recent advancements in vision-language models have enhanced performance by increasing the length of visual tokens, making them much longer than text tokens and significantly raising computational costs. However, we observe that the visual tokens generated by popular vision encoders, such as CLIP and SigLIP, contain significant redundancy. To address this, we introduce VisionZip, a simple yet effective method that selects a set of informative tokens for input to the language model, reducing visual token redundancy and improving efficiency while maintaining model performance. The proposed VisionZip can be widely applied to image and video understanding tasks and is well-suited for multi-turn dialogues in real-world scenarios, where previous methods tend to underperform. Experimental results show that VisionZip outperforms the previous state-of-the-art method by at least 5% performance gains across nearly all settings. Moreover, our method significantly enhances model inference speed, improving the prefilling time by 8x and enabling the LLaVA-Next 13B model to infer faster than the LLaVA-Next 7B model while achieving better results. Furthermore, we analyze the causes of this redundancy and encourage the community to focus on extracting better visual features rather than merely increasing token length. Our code is available at https://github.com/dvlab-research/VisionZip .Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Mobile Edge Intelligence for Large Language Models: A Contemporary SurveyIEEE Communications Surveys & Tutorials, 2025
- ShareGPT4V: Improving Large Multi-modal Models with Better Captions2024
- An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models2024
- MMBench: Is Your Multi-modal Model an All-Around Player?2024
- LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models2024
- Improved Baselines with Visual Instruction Tuning2024
- LISA: Reasoning Segmentation via Large Language Model2024
- MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI2024
- Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks2024
- VLP: A Survey on Vision-language Pre-trainingMachine Intelligence Research, 2023