Preprint
Abstract
Visual Prompt Tuning (VPT) has emerged as a parameter-efficient fine-tuning paradigm for vision transformers, with conventional approaches utilizing dataset-level prompts that remain the same across all input instances. We observe that this strategy results in sub-optimal performance due to high variance in downstream datasets. To address this challenge, we propose Visual Instance-aware Prompt Tuning (ViaPT), which generates instance-aware prompts based on each individual input and fuses them with dataset-level prompts, leveraging Principal Component Analysis (PCA) to retain important prompting information. Moreover, we reveal that VPT-Deep and VPT-Shallow represent two corner cases based on a conceptual understanding, in which they fail to effectively capture instance-specific information, while random dimension reduction on prompts only yields performance between the two extremes. Instead, ViaPT overcomes these limitations by balancing dataset-level and instance-level knowledge, while reducing the amount of learnable parameters compared to VPT-Deep. Extensive experiments across 34 diverse datasets demonstrate that our method consistently outperforms state-of-the-art baselines, establishing a new paradigm for analyzing and optimizing visual prompts for vision transformers.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- CIBR: Cross-Modal Information Bottleneck Regularization for Robust CLIP Generalization2025
- ICH-PRNet: a cross-modal intracerebral haemorrhage prognostic prediction method using joint-attention interaction mechanismNeural Networks, 2025
- E2VPT: An Effective and Efficient Approach for Visual Prompt Tuning2023
- From attribution maps to human-understandable explanations through Concept Relevance PropagationNature Machine Intelligence, 2023
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
- An Empirical Study of Training Self-Supervised Vision Transformers2021
- Deep Residual Learning for Image Recognition2016
- Describing Textures in the Wild2014
- Food-101 – Mining Discriminative Components with Random Forests2014
- Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognitionNeural Networks, 2012