SSPA: Split-and-Synthesize Prompting with Gated Alignments for Multi-Label Image Recognition
Authors:
- Hao TanHTHao Tan
- Zichang TanZTZichang Tan
- Jun LiJLJun Li
- Jun WanJWJun Wan
- Zhen LeiZLZhen Lei
- Stan Z. LiSLStan Z. Li
Preprint
Abstract
Multi-label image recognition is a fundamental task in computer vision. Recently, Vision-Language Models (VLMs) have made notable advancements in this area. However, previous methods fail to effectively leverage the rich knowledge in language models and often incorporate label semantics into visual features unidirectionally. To overcome these problems, we propose a Split-and-Synthesize Prompting with Gated Alignments (SSPA) framework to amplify the potential of VLMs. Specifically, we develop an in-context learning approach to associate the inherent knowledge from LLMs. Then we propose a novel Split-and-Synthesize Prompting (SSP) strategy to first model the generic knowledge and downstream label semantics individually and then aggregate them carefully through the quaternion network. Moreover, we present Gated Dual-Modal Alignments (GDMA) to bidirectionally interact visual and linguistic modalities while eliminating redundant cross-modal information, enabling more efficient region-level alignments. Rather than making the final prediction by a sharp manner in previous works, we propose a soft aggregator to jointly consider results from all image regions. With the help of flexible prompting and gated alignments, SSPA is generalizable to specific domains. Extensive experiments on nine datasets from three domains (i.e., natural, pedestrian attributes and remote sensing) demonstrate the state-of-the-art performance of SSPA. Further analyses verify the effectiveness of SSP and the interpretability of GDMA. The code will be made public.Keywords
All Related Versions
This publication has 36 references indexed in Scilit:
- Improved Baselines with Visual Instruction Tuning2024
- Exploring attribute localization and correlation for pedestrian attribute recognitionNeurocomputing, 2023
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- A Simple Visual-Textual Baseline for Pedestrian Attribute RecognitionIEEE Transactions on Circuits and Systems for Video Technology, 2022
- DRFormer: Learning dual relations using Transformer for pedestrian attribute recognitionNeurocomputing, 2022
- Visual Attention Consistency for Human Attribute RecognitionInternational Journal of Computer Vision, 2022
- Cascaded Split-and-Aggregate Learning with Feature Recombination for Pedestrian Attribute RecognitionInternational Journal of Computer Vision, 2021
- MLRSNet: A multi-label high spatial resolution remote sensing dataset for semantic scene understandingISPRS Journal of Photogrammetry and Remote Sensing, 2020
- Gated Mechanism for Attention Based Multi Modal Sentiment Analysis2020
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009