SAT: Self-Attention Control for Diffusion Models Training
Open Access
- 29 October 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 15-22
Abstract
Recent text-to-image diffusion models show outstanding performance in generating high-quality images conditioned on textual prompts. However, a persistent challenge lies in the generation of detailed images, especially human-related images, which often exhibit distorted faces and eyes. Existing approaches to address this issue either involve the utilization of more specific yet lengthy prompts or the direct application of restoration tools to the generated image. Besides, a few studies have shown that attention maps can enhance diffusion models' stability by guiding intermediate samples during the inference process. In this paper, we propose a novel training strategy (SAT) to improve the sample quality during the training process. To address this issue in a straightforward manner, we introduce blur guidance as a solution to refine intermediate samples, enabling diffusion models to produce higher-quality outputs with a moderate ratio of control. Improving upon this, SAT leverages the intermediate attention maps of diffusion models to further improve training sample quality. Specifically, SAT adversarially blurs only the regions that diffusion models attend to and guide them during the training process. We examine and compare both cross-attention mask control (CAC) and self-attention mask control (SAC) based on stable diffusion (SD) V-1.5, and our results show that our method under SAC (i.e SAT) improves the performance of stable diffusion.Keywords
This publication has 18 references indexed in Scilit:
- Progressive Deblurring of Diffusion Models for Coarse-to-Fine Image SynthesisarXiv, 2022
- Text2HumanACM Transactions on Graphics, 2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Design Guidelines for Prompt Engineering Text-to-Image Generative Models2022
- Hierarchical Text-Conditional Image Generation with CLIP LatentsarXiv, 2022
- Taming Transformers for High-Resolution Image Synthesis2021
- Generative adversarial networksCommunications of the ACM, 2020
- Part-of-Speech Tagging for Twitter with Adversarial Neural Networks2017
- Learning the Compositional Nature of Visual Objects2007
- Learning the Compositional Nature of Objects for Visual Recognition