KNN Transformer with Pyramid Prompts for Few-Shot Learning
Open Access
- 28 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1082-1091
Abstract
Few-Shot Learning (FSL) aims to recognize new classes with limited labeled data. Recent studies have attempted to address the challenge of rare samples with textual prompts to modulate visual features. However, they usually struggle to capture complex semantic relationships between textual and visual features. Moreover, vanilla self-attention is heavily affected by useless information in images, severely constraining the potential of semantic priors in FSL due to the confusion of numerous irrelevant tokens during interaction. To address these aforementioned issues, a K-NN Transformer with Pyramid Prompts (KTPP) is proposed to select discriminative information with K-NN Context Attention (KCA) and adaptively modulate visual features with Pyramid Cross-modal Prompts (PCP). First, for each token, the KCA only selects the K most relevant tokens to compute the self-attention matrix and incorporates the mean of all tokens as the context prompt to provide the global context in three cascaded stages. As a result, irrelevant tokens can be progressively suppressed. Secondly, pyramid prompts are introduced in the PCP to emphasize visual features via interactions between text-based class-aware prompts and multi-scale visual features. This allows the ViT to dynamically adjust the importance weights of visual features based on rich semantic information at different scales, making models robust to spatial variations. Finally, augmented visual features and class-aware prompts are interacted via the KCA to extract class-specific features. Consequently, our model further enhances noise-free visual representations via deep cross-modal interactions, extracting generalized visual representation in scenarios with few labeled samples. Extensive experiments on four benchmark datasets demonstrate significant gains over the state-of-the-art methods, especially for the 1-shot task with 2.28% improvement on average due to semantically enhanced visual representations.Keywords
All Related Versions
Funding Information
- Natural Science Foundation of Shandong Province (ZR2023QF124)
- National Natural Science Foundation of China (U23A20389,62176139)
- Shandong Excellent Young Scientists Fund (Oversea) (2024HWYQ-027)
- Young Scholars Program of Shandong University
- Fundamental Research Funds of Shandong University
This publication has 42 references indexed in Scilit:
- KVT: k-NN Attention for Boosting Vision Transformers2022
- Exploring Plain Vision Transformer Backbones for Object Detection2022
- Visual Prompt Tuning2022
- Self-Promoted Supervision for Few-Shot Transformer2022
- Training Vision Transformers with only 2040 Images2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Few-Shot Learning via Embedding Adaptation With Set-to-Set Functions2020
- DeepEMD: Few-Shot Image Classification With Differentiable Earth Mover’s Distance and Structured Classifiers2020
- Meta-Learning With Differentiable Convex Optimization2019
- ImageNet: A large-scale hierarchical image database2009