A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs
Authors:
- 10 June 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 19814-19824
Abstract
Vision-language models (VLMs) have shown remarkable success across various multi-modal tasks, yet large VLMs encounter significant efficiency challenges due to processing numerous visual tokens. A promising approach to accelerating large VLM inference is using partial information, such as attention maps from specific layers, to assess token importance and prune less essential tokens. However, our study reveals three key insights: (i) Partial attention information is insufficient for accurately identifying critical visual tokens, resulting in suboptimal performance, especially at low token retention ratios; (ii) Global attention information, such as the attention map aggregated across all layers, more effectively preserves essential tokens and maintains comparable performance under aggressive pruning. However, the attention maps from all layers require a full inference pass, which increases computational load and is therefore impractical in existing methods; and (iii) The global attention map aggregated from a small VLM closely resembles that of a large VLM, suggesting an efficient alternative. Based on these findings, we introduce a training-free method, Small VLM Guidance for accelerating Large VLMs (SGL). Specifically, we employ the attention map aggregated from a small VLM to guide visual token pruning in a large VLM. Additionally, an early exiting mechanism is developed to fully use the small VLM’s predictions, dynamically invoking the larger VLM only when necessary, yielding a superior trade-off between accuracy and computation. Extensive evaluations across 11 benchmarks demonstrate the effectiveness and generalizability of SGL, achieving up to 91% pruning ratio for visual tokens while retaining competitive performance. The code is publicly available at https://github.com/NUS-HPC-AI-Lab/SGL.Keywords
All Related Versions
Funding Information
- National Research Foundation
This publication has 27 references indexed in Scilit:
- TransFusion: Generating long, high fidelity time series using diffusion models with transformersMachine Learning with Applications, 2025
- How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suitesScience China Information Sciences, 2024
- An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models2024
- MMBench: Is Your Multi-modal Model an All-Around Player?2024
- LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models2024
- A survey of uncertainty in deep neural networksArtificial Intelligence Review, 2023
- Learning to Weight Samples for Dynamic Early-Exiting Networks2022
- Exploring uncertainty measures in deep networks for Multiple sclerosis lesion detection and segmentationMedical Image Analysis, 2020
- Modeling Context in Referring Expressions2016
- Generation and Comprehension of Unambiguous Object Descriptions2016