Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP
Authors:
Preprint
Abstract
Large pretrained vision-language models like CLIP have shown promising generalization capability, but may struggle in specialized domains (e.g., satellite imagery) or fine-grained classification (e.g., car models) where the visual concepts are unseen or under-represented during pretraining. Prompt learning offers a parameter-efficient finetuning framework that can adapt CLIP to downstream tasks even when limited annotation data are available. In this paper, we improve prompt learning by distilling the textual knowledge from natural language prompts (either human- or LLM-generated) to provide rich priors for those under-represented concepts. We first obtain a prompt ``summary'' aligned to each input image via a learned prompt aggregator. Then we jointly train a prompt generator, optimized to produce a prompt embedding that stays close to the aggregated summary while minimizing task loss at the same time. We dub such prompt embedding as Aggregate-and-Adapted Prompt Embedding (AAPE). AAPE is shown to be able to generalize to different downstream data distributions and tasks, including vision-language understanding tasks (e.g., few-shot classification, VQA) and generation tasks (image captioning) where AAPE achieves competitive performance. We also show AAPE is particularly helpful to handle non-canonical and OOD examples. Furthermore, AAPE learning eliminates LLM-based inference cost as required by baselines, and scales better with data and LLM model size.Keywords
All Related Versions
This publication has 33 references indexed in Scilit:
- MM1: Methods, Analysis and Insights from Multimodal LLM Pre-training2024
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Natural Adversarial Examples2021
- Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks2020
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question AnsweringInternational Journal of Computer Vision, 2018
- Describing Textures in the Wild2014
- Microsoft COCO: Common Objects in Context2014
- Food-101 – Mining Discriminative Components with Random Forests2014
- SUN database: Large-scale scene recognition from abbey to zoo2010
- Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categoriesComputer Vision and Image Understanding, 2007