Training-Free Layout Control with Cross-Attention Guidance
Preprint
Abstract
Recent diffusion-based generators can produce high-quality images from textual prompts. However, they often disregard textual instructions that specify the spatial layout of the composition. We propose a simple approach that achieves robust layout control without the need for training or fine-tuning of the image generator. Our technique manipulates the cross-attention layers that the model uses to interface textual and visual information and steers the generation in the desired direction given, e.g., a user-specified layout. To determine how to best guide attention, we study the role of attention maps and explore two alternative strategies, forward and backward guidance. We thoroughly evaluate our approach on three benchmarks and provide several qualitative examples and a comparative analysis of the two strategies that demonstrate the superiority of backward guidance compared to forward guidance, as well as prior work. We further demonstrate the versatility of layout guidance by extending it to applications such as editing the layout and context of real images.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Adding Conditional Control to Text-to-Image Diffusion Models2023
- BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion2023
- Zero-shot spatial layout conditioning for text-to-image diffusion models2023
- DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation2023
- Imagic: Text-Based Real Image Editing with Diffusion Models2023
- GLIGEN: Open-Set Grounded Text-to-Image Generation2023
- Universal Guidance for Diffusion Models2023
- StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models2015
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015