Predicated Diffusion: Predicate Logic-Based Attention Guidance for Text-to-Image Diffusion Models
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8651-8660
Abstract
Diffusion models have achieved remarkable success in generating high-quality, diverse, and creative images. However, in text-based image generation, they often struggle to accurately capture the intended meaning of the text. For instance, a specified object might not be generated, or an adjective might incorrectly alter unintended objects. Moreover, we found that relationships indicating possession between objects are frequently overlooked. Despite the diversity of users' intentions in text, existing methods often focus on only some aspects of these intentions. In this paper, we propose Predicated Diffusion, a unified framework designed to more effectively express users' intentions. It represents the intended meaning as propositions using predicate logic and treats the pixels in attention maps as fuzzy predicates. This approach provides a differentiable loss function that offers guidance for the image generation process to better fulfill the propositions. Comparative evaluations with existing methods demonstrated that Predicated Diffusion excels in generating images faithful to various text prompts, while maintaining high image quality, as validated by human evaluators and pretrained image-text models.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Training-Free Location-Aware Text-to-Image Synthesis2023
- BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion2023
- Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion ModelsACM Transactions on Graphics, 2023
- Compositional Visual Generation with Composable Diffusion Models2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Assessing Image and Text Generation with Topological Analysis and Fuzzy Logic2021
- Semantic-based regularization for learning and inferenceArtificial Intelligence, 2017
- Theory and Applications of Ordered Fuzzy Numbers2017
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Metamathematics of Fuzzy Logic1998