V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer
Open Access
- 11 April 2025
- journal article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (3) , 3401-3409
Abstract
Concept Bottleneck Models (CBMs) offer inherent interpretability by initially translating images into human-comprehensible concepts, followed by a linear combination of these concepts for classification. However, the annotation of concepts for visual recognition tasks requires extensive expert knowledge and labor, constraining the broad adoption of CBMs. Recent approaches have leveraged the knowledge of large language models to construct concept bottlenecks, with multimodal models like CLIP subsequently mapping image features into the concept feature space for classification. Despite this, the concepts produced by language models can be verbose and may introduce non-visual attributes, which hurts accuracy and interpretability. In this study, we investigate to avoid these issues by constructing CBMs directly from multimodal models. To this end, we adopt common words as base concept vocabulary and leverage auxiliary unlabeled images to construct a Vision-to-Concept (V2C) tokenizer that can explicitly quantize images into their most relevant visual concepts, thus creating a vision-oriented concept bottleneck tightly coupled with the multimodal model. This leads to our V2C-CBM which is training efficient and interpretable with high accuracy. Our V2C-CBM has matched or outperformed LLM-supervised CBMs on various visual classification benchmarks, validating the efficacy of our approach.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Incremental Residual Concept Bottleneck Models2024
- Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantization2023
- Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification2023
- Regularized Vector Quantization for Tokenized Image Synthesis2023
- Explainable Artificial Intelligence (XAI): What we know and what is left to attain Trustworthy Artificial IntelligenceInformation Fusion, 2023
- Autoregressive Image Generation using Residual Quantization2022
- Taming Transformers for High-Resolution Image Synthesis2021
- Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AIInformation Fusion, 2020
- The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesionsScientific Data, 2018
- Describing Textures in the Wild2014