Pedestrian Attribute Recognition via CLIP-Based Prompt Vision-Language Fusion
- 4 September 2024
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 35 (1) , 148-161
Abstract
Existing pedestrian attribute recognition (PAR) algorithms adopt pre-trained CNN (e.g., ResNet) as their backbone network for visual feature learning, which might obtain sub-optimal results due to the insufficient employment of the relations between pedestrian images and attribute labels. In this paper, we formulate PAR as a vision-language fusion problem and fully exploit the relations between pedestrian images and attribute labels. Specifically, the attribute phrases are first expanded into sentences, and then the pre-trained vision-language model CLIP is adopted as our backbone for feature embedding of visual images and attribute descriptions. The contrastive learning objective connects the vision and language modalities well in the CLIP-based feature space, and the Transformer layers used in CLIP can capture the long-range relations between pixels. Then, a multi-modal Transformer is adopted to fuse the dual features effectively and feed-forward network is used to predict attributes. To optimize our network efficiently, we propose the region-aware prompt tuning technique to adjust very few parameters (i.e., only the prompt vectors and classification heads) and fix both the pre-trained VL model and multi-modal Transformer. Our proposed PAR algorithm only adjusts 0.75% learnable parameters compared with the fine-tuning strategy. It also achieves new state-of-the-art performance on both standard and zero-shot settings for PAR, including RAPv1, RAPv2, WIDER, PA100K, and PETA-ZS, RAP-ZS datasets. The source code and pre-trained models will be released on https://github.com/Event-AHU/OpenPAR.Keywords
All Related Versions
Funding Information
- Natural Science Foundation of Anhui Province (2208085J18)
- National Natural Science Foundation of China (62376004, 62102205)
This publication has 64 references indexed in Scilit:
- Tip-Adapter: Training-Free Adaption of CLIP for Few-Shot Classification2022
- Visual Prompt Tuning2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- MCFL: multi-label contrastive focal loss for deep imbalanced pedestrian attribute recognitionNeural Computing and Applications, 2022
- Visual Attention Consistency for Human Attribute RecognitionInternational Journal of Computer Vision, 2022
- Cascaded Split-and-Aggregate Learning with Feature Recombination for Pedestrian Attribute RecognitionInternational Journal of Computer Vision, 2021
- End-to-End Object Detection with Transformers2020
- Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks2020
- Deep Imbalanced Attribute Classification Using Visual Attention Aggregation2018
- Human Attribute Recognition by Deep Hierarchical Contexts2016