Grounded Text-to-Image Synthesis with Attention Refocusing
Authors:
Top Cited Papers
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 7932-7942
Abstract
Driven by the scalable diffusion models trained on large-scale datasets, text-to-image synthesis methods have shown compelling results. However, these models still fail to pre-cisely follow the text prompt involving multiple objects, attributes, or spatial compositions. In this paper, we reveal the potential causes in the diffusion model's cross-attention and self-attention layers. We propose two novel losses to refocus attention maps according to a given spatial layout during sampling. Creating the layouts manually requires additional effort and can be tedious. Therefore, we explore using large language models (LLM) to produce these lay-outs for our method. We conduct extensive experiments on the DrawBench, HRS, and TIFA benchmarks to evaluate our proposed method. We show that our proposed attention re-focusing effectively improves the controllability of existing approaches.Keywords
All Related Versions
This publication has 27 references indexed in Scilit:
- SimpSON: Simplifying Photo Cleanup with Single-Click Distracting Object Segmentation Network2023
- SpaText: Spatio-Textual Representation for Controllable Image Generation2023
- DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation2023
- YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors2023
- Multi-Concept Customization of Text-to-Image Diffusion2023
- GLIGEN: Open-Set Grounded Text-to-Image Generation2023
- Scaling up GANs for Text-to-Image Synthesis2023
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Microsoft COCO: Common Objects in Context2014