Refine, Control and Distill: A Text-to-Image Framework for Faithful Image Generation
- 3 November 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (3) , 2296-2311
Abstract
While text-to-image diffusion models exhibit outstanding results, they struggle to faithfully generate key subjects with corresponding attributes in prompts, challenges known as catastrophic neglect and attribute binding. Previous works typically utilize attention adjustments to solve the above problems, whereas we observe that they may still generate unfaithful images. In this paper, we carefully analyze the text-to-image process and pinpoint three pivotal bottlenecks that hinder image faithful generation: (1) unequal responses of neglected subjects in text embedding, (2) competition and entanglement between subjects’ attention, and (3) suboptimal quality of intermediate features from U-Net. Based on the aforementioned observations, we propose a Refine, Control, and Distill (RCD) framework built upon the stable diffusion model to alleviate the negative effects raised by the bottlenecks mentioned above, respectively. Specifically, we achieve the above goals through a text embedding refinement module, three region-level attention control losses, and self-distillation of intermediate semantic features in the denoising process. Our approach exhibits promising capability in generating faithful and high-quality images and outperforms state-of-the-art methods through extensive quantitative and qualitative evaluations on recent advanced base diffusion models.Keywords
Funding Information
- National Natural Science Foundation of China (62425603)
- Basic Research Program of Jiangsu Province (BK20240011)
This publication has 38 references indexed in Scilit:
- Be Yourself: Bounded Attention for Multi-subject Text-to-Image Generation2024
- Multi-Concept Customization of Text-to-Image Diffusion2023
- Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation2023
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors2022
- Compositional Visual Generation with Composable Diffusion Models2022
- No Token Left Behind: Explainability-Aided Image Classification and Generation2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Blended Diffusion for Text-driven Editing of Natural Images2022
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks2018
- Microsoft COCO: Common Objects in Context2014