Spatial Transport Optimization by Repositioning Attention Map for Training-Free Text-to-Image Synthesis
Authors:
- Yeonkyung LeeYLYeonkyung Lee
- Chanyoung KimCKChanyoung Kim
- Kwanghyun ParkKPKwanghyun Park
Preprint
Abstract
Diffusion-based text-to-image (T2I) models have recently excelled in high-quality image generation, particularly in a training-free manner, enabling cost-effective adaptability and generalization across diverse tasks. However, while the existing methods have been continuously focusing on several challenges, such as "missing objects" and "mismatched attributes," another critical issue of "mislocated objects" remains where generated spatial positions fail to align with text prompts. Surprisingly, ensuring such seemingly basic functionality remains challenging in popular T2I models due to the inherent difficulty of imposing explicit spatial guidance via text forms. To address this, we propose STORM (Spatial Transport Optimization by Repositioning Attention Map), a novel training-free approach for spatially coherent T2I synthesis. STORM employs Spatial Transport Optimization (STO), rooted in optimal transport theory, to dynamically adjust object attention maps for precise spatial adherence, supported by a Spatial Transport (ST) Cost function that enhances spatial understanding. Our analysis shows that integrating spatial awareness is most effective in the early denoising stages, while later phases refine details. Extensive experiments demonstrate that STORM surpasses existing methods, effectively mitigating mislocated objects while improving missing and mismatched attributes, setting a new benchmark for spatial alignment in T2I synthesis.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- REVISION: Rendering Tools Enable Spatial Fidelity in Vision-Language Models2024
- Advancing Text-Driven Chest X-Ray Generation with Policy-Based Reinforcement Learning2024
- Getting it Right: Improving Spatial Consistency in Text-to-Image Models2024
- Explaining generative diffusion models via visual analysis for interpretable decision-making processExpert Systems with Applications, 2024
- Simple Open-Vocabulary Object Detection2022
- Compositional Visual Generation with Composable Diffusion Models2022
- Simple Multi-dataset Detection2022
- Semantic Image Synthesis With Spatially-Adaptive Normalization2019
- Image-to-Image Translation with Conditional Adversarial Networks2017
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015