Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspective
Preprint
Abstract
The rapid development of large-scale deep learning models questions the affordability of hardware platforms, which necessitates the pruning to reduce their computational and memory footprints. Sparse neural networks as the product, have demonstrated numerous favorable benefits like low complexity, undamaged generalization, etc. Most of the prominent pruning strategies are invented from a model-centric perspective, focusing on searching and preserving crucial weights by analyzing network topologies. However, the role of data and its interplay with model-centric pruning has remained relatively unexplored. In this research, we introduce a novel data-model co-design perspective: to promote superior weight sparsity by learning important model topology and adequate input data in a synergetic manner. Specifically, customized Visual Prompts are mounted to upgrade neural Network sparsification in our proposed VPNs framework. As a pioneering effort, this paper conducts systematic investigations about the impact of different visual prompts on model pruning and suggests an effective joint optimization approach. Extensive experiments with 3 network architectures and 8 datasets evidence the substantial performance improvements from VPNs over existing start-of-the-art pruning algorithms. Furthermore, we find that subnetworks discovered by VPNs from pre-trained models enjoy better transferability across diverse downstream scenarios. These insights shed light on new promising possibilities of data-model co-designs for vision model sparsification.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Enhancing Adversarial Training via Reweighting Optimization Trajectory2023
- Dynamic Sparse Training via Balancing the Exploration-Exploitation Trade-off2023
- Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language ProcessingACM Computing Surveys, 2023
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks2022
- The Power of Scale for Parameter-Efficient Prompt Tuning2021
- How Can We Know What Language Models Know?Transactions of the Association for Computational Linguistics, 2020
- AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts2020
- Improving model selection by nonconvergent methodsNeural Networks, 1993