LoMOE: Localized Multi-Object Editing via Multi-Diffusion
Open Access
- 28 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 3342-3351
Abstract
Recent developments in diffusion models have demonstrated an exceptional capacity to generate high-quality, prompt-conditioned image edits. Nevertheless, previous approaches have primarily relied on textual prompts for image editing, which tend to be less effective when making precise edits to specific objects or fine-grained regions within a scene containing single/multiple objects. We introduce a novel framework for zero-shot localized multi-object editing through a multi-diffusion process to overcome this challenge. This framework empowers users to perform various operations on objects within an image, such as adding, replacing, or editing many objects in a complex scene in one pass. Our approach leverages foreground masks and corresponding simple text prompts that exert localized influences on the target regions resulting in high-fidelity image editing. A combination of cross-attention and background preservation losses within the latent space ensures that the characteristics of the object being edited are preserved while simultaneously achieving a high-quality, seamless reconstruction of the background with fewer artifacts compared to the state-of-the-art (SOTA). We also curate and release a dataset dedicated to multi-object editing, named LoMOE-Bench. Our experiments against existing SOTA demonstrate the improved effectiveness of our approach in terms of both image editing quality, and inference speed.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Null-text Inversion for Editing Real Images using Guided Diffusion Models2023
- Imagic: Text-Based Real Image Editing with Diffusion Models2023
- Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation2023
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors2022
- No Token Left Behind: Explainability-Aided Image Classification and Generation2022
- VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- StarGAN v2: Diverse Image Synthesis for Multiple Domains2020
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- Structural matching in computer vision using probabilistic relaxationIEEE Transactions on Pattern Analysis and Machine Intelligence, 1995