TIAM - A Metric for Evaluating Alignment in Text-to-Image Generation
Authors:
- 3 January 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2878-2887
Abstract
The progress in the generation of synthetic images has made it crucial to assess their quality. While several metrics have been proposed to assess the rendering of images, it is crucial for Text-to-Image (T2I) models, which generate images based on a prompt, to consider additional aspects such as to which extent the generated image matches the important content of the prompt. Moreover, although the generated images usually result from a random starting point, the influence of this one is generally not considered. In this article, we propose a new metric based on prompt templates to study the alignment between the content specified in the prompt and the corresponding generated images. It allows us to better characterize the alignment in terms of the type of the specified objects, their number, and their color. We conducted a study on several recent T2I models about various aspects. An additional interesting result we obtained with our approach is that image quality can vary drastically depending on the noise used as a seed for the images. We also quantify the influence of the number of concepts in the prompt, their order as well as their (color) attributes. Finally, our method allows us to identify some seeds that produce better images than others, opening novel directions of research on this understudied topic.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- A taxonomy of prompt modifiers for text-to-image generationBehaviour & Information Technology, 2023
- Linear Spaces of Meanings: Compositional Structures in Vision-Language Models2023
- Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion ModelsACM Transactions on Graphics, 2023
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Semantic Object Accuracy for Generative Text-to-Image SynthesisIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
- Pros and cons of GAN evaluation measuresComputer Vision and Image Understanding, 2019
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- Microsoft COCO: Common Objects in Context2014
- Ensemble Methods in Machine Learning2000
- Measuring nominal scale agreement among many raters.Psychological Bulletin, 1971