BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning
Authors:
- Changdae OhCOChangdae Oh
- Jiyoung JungJJJiyoung Jung
- Kyungwoo SongKSKyungwoo Song
Preprint
Abstract
With the surge of large-scale pre-trained models (PTMs), fine-tuning these models to numerous downstream tasks becomes a crucial problem. Consequently, parameter efficient transfer learning (PETL) of large models has grasped huge attention. While recent PETL methods showcase impressive performance, they rely on optimistic assumptions: 1) the entire parameter set of a PTM is available, and 2) a sufficiently large memory capacity for the fine-tuning is equipped. However, in most real-world applications, PTMs are served as a black-box API or proprietary software without explicit parameter accessibility. Besides, it is hard to meet a large memory requirement for modern PTMs. In this work, we propose black-box visual prompting (BlackVIP), which efficiently adapts the PTMs without knowledge about model architectures and parameters. BlackVIP has two components; 1) Coordinator and 2) simultaneous perturbation stochastic approximation with gradient correction (SPSA-GC). The Coordinator designs input-dependent image-shaped visual prompts, which improves few-shot adaptation and robustness on distribution/location shift. SPSA-GC efficiently estimates the gradient of a target model to update Coordinator. Extensive experiments on 16 datasets demonstrate that BlackVIP enables robust adaptation to diverse domains without accessing PTMs' parameters, with minimal memory requirements. Code: \url{https://github.com/changdaeoh/BlackVIP}Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Unleashing Vanilla Vision Transformer with Masked Image Modeling for Object Detection2023
- CLIP-Adapter: Better Vision-Language Models with Feature AdaptersInternational Journal of Computer Vision, 2023
- MaPLe: Multi-modal Prompt Learning2023
- Prompting Visual-Language Models for Efficient Video Understanding2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Random Gradient-Free Minimization of Convex FunctionsFoundations of Computational Mathematics, 2015
- Introduction to Stochastic Search and Optimization2003
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- A one-measurement form of simultaneous perturbation stochastic approximationAutomatica, 1997
- Multivariate stochastic approximation using a simultaneous perturbation gradient approximationIEEE Transactions on Automatic Control, 1992