Training-Free Layout Control with Cross-Attention Guidance
Authors:
Top Cited Papers
- 3 January 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 5331-5341
Abstract
Recent diffusion-based generators can produce high-quality images from textual prompts. However, they often disregard textual instructions that specify the spatial layout of the composition. We propose a simple approach that achieves robust layout control without the need for training or fine-tuning of the image generator. Our technique manipulates the cross-attention layers that the model uses to interface textual and visual information and steers the generation in the desired direction given, e.g., a user-specified layout. To determine how to best guide attention, we study the role of attention maps and explore two alternative strategies, forward and backward guidance. We thoroughly evaluate our approach on three benchmarks and provide several qualitative examples and a comparative analysis of the two strategies that demonstrate the superiority of backward guidance compared to forward guidance, as well as prior work. We further demonstrate the versatility of layout guidance by extending it to applications such as editing the layout and context of real images.Keywords
All Related Versions
This publication has 36 references indexed in Scilit:
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors2022
- Trace Controlled Text to Image Generation2022
- Compositional Visual Generation with Composable Diffusion Models2022
- Multimodal Conditional Image Synthesis with Product-of-Experts GANs2022
- Semantic Image Synthesis With Spatially-Adaptive Normalization2019
- Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis2018
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks2018
- Image Generation from Scene Graphs2018
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Microsoft COCO: Common Objects in Context2014