BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 24224-24235
Abstract
With the surge of large-scale pre-trained models (PTMs), fine-tuning these models to numerous downstream tasks becomes a crucial problem. Consequently, parameter efficient transfer learning (PETL) of large models has grasped huge attention. While recent PETL methods showcase impressive performance, they rely on optimistic assumptions: 1) the entire parameter set of a PTM is available, and 2) a sufficiently large memory capacity for the fine-tuning is equipped. However, in most real-world applications, PTMs are served as a black-box API or proprietary software without explicit parameter accessibility. Besides, it is hard to meet a large memory requirement for modern PTMs. In this work, we propose black-box visual prompting (Black-VIP), which efficiently adapts the PTMs without knowledge about model architectures and parameters. Black-VIP has two components; 1) Coordinator and 2) simultaneous perturbation stochastic approximation with gradient correction (SPSA-GC). The Coordinator designs inputdependent image-shaped visual prompts, which improves few-shot adaptation and robustness on distribution/location shift. SPSA-GC efficiently estimates the gradient of a target model to update Coordinator. Extensive experiments on 16 datasets demonstrate that BlackVIP enables robust adaptation to diverse domains without accessing PTMs' parameters, with minimal memory requirements. Code: https://github.com/changdaeoh/BlackVIPKeywords
All Related Versions
Funding Information
- National Research Foundation of Korea (2022R1A4A3033874,2021R1F1A1060117,2020R1C1C1008726,2017R1D1A1 B05028565)
This publication has 42 references indexed in Scilit:
- Deep Residual Learning for Image Recognition2016
- The Limitations of Deep Learning in Adversarial Settings2016
- Robust Neural Network Tracking Controller Using Simultaneous Perturbation Stochastic ApproximationIEEE Transactions on Neural Networks, 2008
- Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES)Evolutionary Computation, 2003
- Accelerated second-order stochastic optimization using only function measurements2002
- Completely Derandomized Self-Adaptation in Evolution StrategiesEvolutionary Computation, 2001
- Adaptive stochastic approximation by the simultaneous perturbation methodIEEE Transactions on Automatic Control, 2000
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- A one-measurement form of simultaneous perturbation stochastic approximationAutomatica, 1997
- Multivariate stochastic approximation using a simultaneous perturbation gradient approximationIEEE Transactions on Automatic Control, 1992