Diffusion Model is Secretly a Training-Free Open Vocabulary Semantic Segmenter
Top Cited Papers
- 24 March 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 34 (10577149) , 1895-1907
Abstract
The pre-trained text-image discriminative models, such as CLIP, has been explored for open-vocabulary semantic segmentation with unsatisfactory results due to the loss of crucial localization information and awareness of object shapes. Recently, there has been a growing interest in expanding the application of generative models from generation tasks to semantic segmentation. These approaches utilize generative models either for generating annotated data or extracting features to facilitate semantic segmentation. This typically involves generating a considerable amount of synthetic data or requiring additional mask annotations. To this end, we uncover the potential of generative text-to-image diffusion models (e.g., Stable Diffusion) as highly efficient open-vocabulary semantic segmenters, and introduce a novel training-free approach named DiffSegmenter. The insight is that to generate realistic objects that are semantically faithful to the input text, both the complete object shapes and the corresponding semantics are implicitly learned by diffusion models. We discover that the object shapes are characterized by the self-attention maps while the semantics are indicated through the cross-attention maps produced by the denoising U-Net, forming the basis of our segmentation results. Additionally, we carefully design effective textual prompts and a category filtering mechanism to further enhance the segmentation results. Extensive experiments on three benchmark datasets show that the proposed DiffSegmenter achieves impressive results for open-vocabulary semantic segmentation.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62461160331, 62132001)
- Natural Science Foundation of China/Research Grants Council (RGC) Collaborative Research Scheme (CRS_HKU703/24)
- Hong Kong RGC General Research Fund (17203023)
- Hong Kong RGC Collaborative Research Fund (C5052-23G)
- Hong Kong Jockey Club Charities Trust (2022-0174)
- UBTECH Robotics
- Fundamental Research Funds for the Central Universities
- Young Elite Scientists Sponsorship Program by China Association for Science and Technology
- Beihang World TOP University Collaboration Program
This publication has 41 references indexed in Scilit:
- Object-Conditioned Energy-Based Attention Map Alignment in Text-to-Image Diffusion Models2024
- Open-Vocabulary SAM: Segment and Recognize Twenty-Thousand Classes Interactively2024
- Cold SegDiffusion: A novel diffusion model for medical image segmentationKnowledge-Based Systems, 2024
- Scaling Open-Vocabulary Image Segmentation with Image-Level Labels2022
- Compositional Visual Generation with Composable Diffusion Models2022
- Open-World Semantic Segmentation via Contrasting and Clustering Vision-Language Embedding2022
- Extract Free Dense Labels from CLIP2022
- Variational Inference: A Review for StatisticiansJournal of the American Statistical Association, 2017
- Microsoft COCO: Common Objects in Context2014
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009