Exploring Effective Factors for Improving Visual In-Context Learning
Top Cited Papers
- 31 March 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 34 (10577149) , 2147-2160
Abstract
The In-Context Learning (ICL) is to understand a new task via a few demonstrations (aka. prompt) and predict new inputs without tuning the models. While it has been widely studied in NLP, it is still a relatively new area of research in computer vision. To reveal the factors influencing the performance of visual in-context learning, this paper shows that Prompt Selection and Prompt Fusion are two major factors that have a direct impact on the inference performance of visual in-context learning. Prompt selection is the process of selecting the most suitable prompt for query image. This is crucial because high-quality prompts assist large-scale visual models in rapidly and accurately comprehending new tasks. Prompt fusion involves combining prompts and query images to activate knowledge within large-scale visual models. However, altering the prompt fusion method significantly impacts its performance on new tasks. Based on these findings, we propose a simple framework prompt-SelF to improve visual in-context learning. Specifically, we first use the pixel-level retrieval method to select a suitable prompt, and then use different prompt fusion methods to activate diverse knowledge stored in the large-scale vision model, and finally, ensemble the prediction results obtained from different prompt fusion methods to obtain the final prediction results. We conducted extensive experiments on single-object segmentation and detection tasks to demonstrate the effectiveness of prompt-SelF. Remarkably, prompt-SelF has outperformed OSLSM method-based meta-learning in 1-shot segmentation for the first time. This indicated the great potential of visual in-context learning. The source code and models will be available at https://github.com/syp2ysy/prompt-SelF.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62425603)
- Basic Research Program of Jiangsu Province (BK20240011, BK20243018)
This publication has 46 references indexed in Scilit:
- Context Autoencoder for Self-supervised Representation LearningInternational Journal of Computer Vision, 2023
- Self-support Few-Shot Semantic Segmentation2022
- Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation2022
- Dynamic Prototype Convolution Network for Few-Shot Semantic Segmentation2022
- Learning What Not to Segment: A New Perspective on Few-Shot Segmentation2022
- Adaptive Prototype Learning and Allocation for Few-Shot Segmentation2021
- FSS-1000: A 1000-Class Dataset for Few-Shot Segmentation2020
- Microsoft COCO: Common Objects in Context2014
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009
- ImageNet: A large-scale hierarchical image database2009