Pedestrian Attribute Recognition via CLIP based Prompt Vision-Language Fusion
Preprint
Abstract
Existing pedestrian attribute recognition (PAR) algorithms adopt pre-trained CNN (e.g., ResNet) as their backbone network for visual feature learning, which might obtain sub-optimal results due to the insufficient employment of the relations between pedestrian images and attribute labels. In this paper, we formulate PAR as a vision-language fusion problem and fully exploit the relations between pedestrian images and attribute labels. Specifically, the attribute phrases are first expanded into sentences, and then the pre-trained vision-language model CLIP is adopted as our backbone for feature embedding of visual images and attribute descriptions. The contrastive learning objective connects the vision and language modalities well in the CLIP-based feature space, and the Transformer layers used in CLIP can capture the long-range relations between pixels. Then, a multi-modal Transformer is adopted to fuse the dual features effectively and feed-forward network is used to predict attributes. To optimize our network efficiently, we propose the region-aware prompt tuning technique to adjust very few parameters (i.e., only the prompt vectors and classification heads) and fix both the pre-trained VL model and multi-modal Transformer. Our proposed PAR algorithm only adjusts 0.75% learnable parameters compared with the fine-tuning strategy. It also achieves new state-of-the-art performance on both standard and zero-shot settings for PAR, including RAPv1, RAPv2, WIDER, PA100K, and PETA-ZS, RAP-ZS datasets. The source code and pre-trained models will be released on https://github.com/Event-AHU/OpenPAR.Keywords
All Related Versions
This publication has 27 references indexed in Scilit:
- CPT: Colorful Prompt Tuning for pre-trained vision-language modelsAI Open, 2024
- CLIP-Adapter: Better Vision-Language Models with Feature AdaptersInternational Journal of Computer Vision, 2023
- Large-scale Multi-modal Pre-trained Models: A Comprehensive SurveyMachine Intelligence Research, 2023
- Diverse features discovery transformer for pedestrian attribute recognitionEngineering Applications of Artificial Intelligence, 2022
- Visual Prompt Tuning2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Visual Attention Consistency for Human Attribute RecognitionInternational Journal of Computer Vision, 2022
- Cascaded Split-and-Aggregate Learning with Feature Recombination for Pedestrian Attribute RecognitionInternational Journal of Computer Vision, 2021
- Time series forecasting using artificial neural networks methodologies: A systematic reviewFuture Computing and Informatics Journal, 2018
- Deep Imbalanced Attribute Classification Using Visual Attention Aggregation2018