AG-CLIP: Attribute-Guided CLIP for Zero-Shot Fine-Grained Recognition
Open Access
- 15 January 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Open Journal of the Computer Society
- Vol. 7, 365-375
Abstract
Zero-shot fine-grained recognition is challenging due to high visual similarities between classes and the inferior encoding of fine-grained features in embedding models. In this work, we present an attribute-guided Contrastive Language-Image Pre-training (AG-CLIP) model with an additional attribute encoder. Our approach first identifies relevant visual attributes from the textual class descriptions using an attribute mining module leveraging a large language model (LLM) GPT-4o. The attributes are then used to construct prompts for an open vocabulary object/region detector to extract relevant corresponding image regions. The attribute text, along with focused regions of the input, then guides the CLIP model to focus on these discriminative attributes during fine-tuning through a context-attribute fusion module. Our attribute-guided attention mechanism allows CLIP to effectively disambiguate fine-grained classes by highlighting their distinctive attributes without requiring fine-tuning or additional training data on unseen classes. We evaluate our approach on the CUB-200-2011 and plant disease datasets, achieving 73.3% and 84.6% accuracy, respectively. Our method achieves state-of-the-art zero-shot performance, outperforming prior methods that rely on external knowledge bases or complex meta-learning strategies. The strong results demonstratethe effectiveness of injecting generic attribute awareness into powerful vision-language models like CLIP for tackling fine-grained recognition in a zero-shot manner.Keywords
Funding Information
- UAE University Startup Research (12T095)
- UAE University Startup Research (G00005050)
This publication has 44 references indexed in Scilit:
- Leveraging model explainability and fine-grained cutmix augmentation for robust detection of apricot diseases in UAV imagesExpert Systems with Applications, 2025
- Leveraging Large Language Models and Weak Supervision for Social Media Data Annotation: An Evaluation Using COVID-19 Self-reported Vaccination Tweets2023
- ChatGPT outperforms crowd workers for text-annotation tasksProceedings of the National Academy of Sciences of the United States of America, 2023
- FieldPlant: A Dataset of Field Plant Images for Plant Disease Detection and Classification With Deep LearningIEEE Access, 2023
- Simple Open-Vocabulary Object Detection2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- A comprehensive guava leaves and fruits dataset for guava disease recognitionData in Brief, 2022
- Region Graph Embedding Network for Zero-Shot Learning2020
- Zero-Shot Visual Recognition Using Semantics-Preserving Adversarial Embedding Networks2018
- Exploiting Linked Data and Knowledge Graphs in Large Organisations2017