Text to Image for Multi-Label Image Recognition With Joint Prompt-Adapter Learning
- 26 May 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47 (9) , 7660-7674
Abstract
Benefited from image-text contrastive learning, pre-trained vision-language models, e.g., CLIP, allow to direct leverage texts as images (TaI) for parameter-efficient fine-tuning (PEFT). While CLIP is capable of making image features to be similar to the corresponding text features, the modality gap remains a nontrivial issue and limits image recognition performance of TaI. Using multi-label image recognition (MLR) as an example, we present a novel method, called T2I-PAL to tackle the modality gap issue when using only text captions for PEFT. The core design of T2I-PAL is to leverage pre-trained text-to-image generation models to generate photo-realistic and diverse images from text captions, thereby reducing the modality gap. To further enhance MLR, T2I-PAL incorporates a class-wise heatmap and learnable prototypes. This aggregates local similarities, making the representation of local visual features more robust and informative for multi-label recognition. For better PEFT, we further combine both prompt tuning and adapter learning to enhance classification performance. T2I-PAL offers significant advantages: it eliminates the need for fully semantically annotated training images, thereby reducing the manual annotation workload, and it preserves the intrinsic mode of the CLIP model, allowing for seamless integration with any existing CLIP framework. Extensive experiments on multiple benchmarks, including MS-COCO, VOC2007, and NUS-WIDE, show that our T2I-PAL can boost recognition performance by 3.47% in average above the top-ranked state-of-the-art methods.Keywords
All Related Versions
Funding Information
- Agency for Science, Technology and Research
- AME Programmatic Funding Scheme (A20H4b0141)
- National Research Foundation (NRF) Singapore
- AI Singapore Programme AISG (AISG2-TC-2021-003)
- Agency for Science, Technology and Research
- RIE2020 Health and Biomedical Sciences
- Industry Alignment Fund Pre-Positioning (H20C6a0032)
- A*STAR Central Research Fund
This publication has 35 references indexed in Scilit:
- Class Concept Representation From Contextual Texts for Training-Free Multi-Label RecognitionIEEE Access, 2025
- Tip-Adapter: Training-Free Adaption of CLIP for Few-Shot Classification2022
- Compositional Visual Generation with Composable Diffusion Models2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- LaSO: Label-Set Operations Networks for Multi-Label Few-Shot Learning2019
- Learning a Deep ConvNet for Multi-Label Classification With Partial Labels2019
- Multi-Label Image Recognition With Graph Convolutional Networks2019
- CNN-RNN: A Unified Framework for Multi-label Image Classification2016
- Microsoft COCO: Common Objects in Context2014
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009