ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models
Preprint
Abstract
Large Vision-Language Models (LVLMs) can understand the world comprehensively by integrating rich information from different modalities, achieving remarkable advancements on various multimodal downstream tasks. However, deploying LVLMs is often problematic due to their massive computational/energy costs and carbon consumption. Such issues make it infeasible to adopt conventional iterative global pruning, which is costly due to computing the Hessian matrix of the entire large model for sparsification. Alternatively, several studies have recently proposed layer-wise pruning approaches to avoid the expensive computation of global pruning and efficiently compress model weights according to their importance within a layer. However, they often suffer from suboptimal model compression due to their lack of a global perspective. To address this limitation in recent efficient pruning methods for large models, we propose Efficient Coarse-to-Fine LayerWise Pruning (ECoFLaP), a two-stage coarse-to-fine weight pruning approach for LVLMs. We first determine the sparsity ratios of different layers or blocks by leveraging the global importance score, which is efficiently computed based on the zeroth-order approximation of the global model gradients. Then, the model performs local layer-wise unstructured weight pruning based on globally-informed sparsity ratios. We validate our proposed method across various multimodal and unimodal models and datasets, demonstrating significant performance improvements over prevalent pruning techniques in the high-sparsity regime.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Otter: A Multi-Modal Model With In-Context Instruction TuningIEEE Transactions on Pattern Analysis and Machine Intelligence, 2025
- Improved Baselines with Visual Instruction Tuning2024
- BiTAT: Neural Network Binarization with Task-Dependent Aggregated Transformation2023
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- On the effect of dropping layers of pre-trained transformer modelsComputer Speech & Language, 2022
- EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover ClassificationIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019
- Learning visual and textual representations for multimodal matching and classificationPattern Recognition, 2018
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question AnsweringInternational Journal of Computer Vision, 2018
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence ModelsInternational Journal of Computer Vision, 2016
- Deep compression and EIE: Efficient inference engine on compressed deep neural network2016