Optimization of Prompt Learning via Multi-Knowledge Representation for Vision-Language Models
- 14 August 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 7557-7569
Abstract
Vision-language models (VLMs), such as CLIP, play a foundational role in various cross-modal applications. To fully leverage the potential of VLMs in adapting to downstream tasks, context optimization methods such as prompt tuning are essential. However, one key limitation is the lack of diversity in prompt templates, whether they are hand-crafted or learned through additional modules. This limitation restricts the capabilities of pretrained VLMs and can result in incorrect predictions in downstream tasks. To address this challenge, we propose context optimization with multi-knowledge representation (CoKnow), a framework that enhances prompt learning for VLMs with rich contextual knowledge. To facilitate CoKnow during inference, we train lightweight semantic knowledge mappers, which are capable of generating multi-knowledge representations for an input image without requiring additional priors. Experimentally, we conduct extensive experiments on 11 publicly available datasets, demonstrating that CoKnow outperforms a series of previous methods.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62271485)
- Beijing Municipal Science and Technology Project (Z231100007423004)
This publication has 32 references indexed in Scilit:
- Context-Aware Robust Fine-TuningInternational Journal of Computer Vision, 2023
- CLIP-Adapter: Better Vision-Language Models with Feature AdaptersInternational Journal of Computer Vision, 2023
- Tip-Adapter: Training-Free Adaption of CLIP for Few-Shot Classification2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences, 2017
- CIFAR-10: KNN-Based Ensemble of Classifiers2016
- Food-101 – Mining Discriminative Components with Random Forests2014
- SUN database: Large-scale scene recognition from abbey to zoo2010
- ImageNet: A large-scale hierarchical image database2009
- Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories2005