AMITA: Attribute-Guided Masked Image-Text Alignment for Multi-Label Image Representation
- 6 June 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 35 (11) , 11432-11447
Abstract
Multi-label image classification, which involves recognizing multiple objects within a single image, is a fundamental task in computer vision. Recently, Visual-Language Models (VLMs) have made remarkable progress in this area. Many approaches combine textual and visual modalities to understand the entire image. In this paper, we find that there is a direct correlation between the accurate localization of objects and the accuracy of multi-label classification. However, previous research methods did not specifically address localization accuracy, resulting in sub-optimal accuracy. Therefore, we propose the AMITA, namely Attribute-guided Masked Image-Text Alignment for multi-label image representation. AMITA improves localization accuracy by segmenting object masks, thereby enhancing the accuracy of multi-label image classification. Additionally, AMITA introduces an AutoFocus method to handle the localization problem of small objects. AutoFocus conducts recognition by resizing and cropping the image respectively, and automatically selects the images useful for the classification target. Moreover, AMITA incorporates Attribute-guided Prompting to strengthen the semantic distinction among different categories. It uses large language models to obtain the attributes of different categories and carefully designs prompts to enhance the attribute differences among different categories. Finally, extensive experiments on three popular datasets, including MS-COCO, Pascal VOC 2007, and NUS-WIDE, demonstrate the superiority of AMITA.Keywords
Funding Information
- National Natural Science Foundation of China (62472332)
- Hubei Provincial Undergraduate Higher Education Provincial Level Teaching Reform Research Project (2024130)
This publication has 66 references indexed in Scilit:
- Triplet attention and dual-pool contrastive learning for clinic-driven multi-label medical image classificationMedical Image Analysis, 2023
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- A multi-scale semantic attention representation for multi-label image recognition with graph networksNeurocomputing, 2022
- Double Attention for Multi-Label Image ClassificationIEEE Access, 2020
- Attention-Driven Dynamic Graph Convolutional Network for Multi-label Image Recognition2020
- Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based LocalizationInternational Journal of Computer Vision, 2019
- Combining local and global hypotheses in deep neural network for multi-label image classificationNeurocomputing, 2017
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Microsoft COCO: Common Objects in Context2014
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009