Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspective
Authors:
Open Access
- 11 April 2025
- journal article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (4) , 4111-4119
Abstract
The rapid development of large-scale deep learning models questions the affordability of hardware platforms, which necessitates the pruning to reduce their computational and memory footprints. Sparse neural networks as the product, have demonstrated numerous favorable benefits like low complexity, undamaged generalization, etc. Most of the prominent pruning strategies are invented from a model-centric perspective, focusing on searching and preserving crucial weights by analyzing network topologies. However, the role of data and its interplay with model-centric pruning has remained relatively unexplored. In this research, we introduce a novel data-model co-design perspective: to promote superior weight sparsity by learning important model topology and adequate input data in a synergetic manner. Specifically, customized Visual Prompts are mounted to upgrade neural Network sparsification in our proposed VPNs framework. As a pioneering effort, this paper conducts systematic investigations about the impact of different visual prompts on model pruning and suggests an effective joint optimization approach. Extensive experiments with 3 network architectures and 8 datasets evidence the substantial performance improvements from VPNs over existing start-of-the-art pruning algorithms. Furthermore, we find that subnetworks discovered by VPNs from pre-trained models enjoy better transferability across diverse downstream scenarios. These insights shed light on new promising possibilities of data-model co-designs for vision model sparsification.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language ProcessingACM Computing Surveys, 2023
- LLM.int8(): 8-Bit Matrix Multiplication for Transformers at Scale2022
- P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks2022
- Block Pruning For Faster Transformers2021
- The Power of Scale for Parameter-Efficient Prompt Tuning2021
- Prefix-Tuning: Optimizing Continuous Prompts for Generation2021
- Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network scienceNature Communications, 2018
- Improving model selection by nonconvergent methodsNeural Networks, 1993
- Using Relevance to Reduce Network Size AutomaticallyConnection Science, 1989