Localizing Object-level Shape Variations with Text-to-Image Diffusion Models
Authors:
Top Cited Papers
- 1 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 22994-23004
Abstract
Text-to-image models give rise to workflows which often begin with an exploration step, where users sift through a large collection of generated images. The global nature of the text-to-image generation process prevents users from narrowing their exploration to a particular object in the image. In this paper, we present a technique to generate a collection of images that depicts variations in the shape of a specific object, enabling an object-level shape exploration process. Creating plausible variations is challenging as it requires control over the shape of the generated object while respecting its semantics. A particular challenge when generating object variations is accurately localizing the manipulation applied over the object’s shape. We introduce a prompt-mixing technique that switches between prompts along the denoising process to attain a variety of shape choices. To localize the image-space operation, we present two techniques that use the self-attention layers in conjunction with the cross-attention layers. Moreover, we show that these localization techniques are general and effective beyond the scope of generating object variations. Extensive results and comparisons demonstrate the effectiveness of our method in generating object variations, and the competence of our localization techniques.Keywords
All Related Versions
Funding Information
- Israel Science Foundation
This publication has 30 references indexed in Scilit:
- Text2LIVE: Text-Driven Layered Image and Video Editing2022
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- MaskGIT: Masked Generative Image Transformer2022
- Image Segmentation Using Text and Image Prompts2022
- DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort2021
- Interpreting the Latent Space of GANs for Semantic Face Editing2020
- Editing in Style: Uncovering the Local Semantics of GANs2020
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks2018
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018