An Item Is Worth a Prompt: Versatile Image Editing with Disentangled Control
Open Access
- 11 April 2025
- journal article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (16) , 16559-16567
Abstract
Building on the success of text-to-image diffusion models (DPMs), image editing is an important application to enable human interaction with AI-generated content. Among various editing methods, editing within the prompt space gains more attention due to its capacity and simplicity of controlling semantics. However, since diffusion models are commonly pretrained on descriptive text captions, direct editing of words in text prompts usually leads to completely different generated images, violating the requirements for image editing. On the other hand, existing editing methods usually consider introducing spatial masks to preserve the identity of unedited regions, which are usually ignored by DPMs and therefore lead to inharmonic editing results. Targeting these two challenges, in this work, we propose to disentangle the comprehensive image-prompt interaction into several item-prompt interactions, with each item linked to a special learned prompt. The resulting framework, named D-Edit, is based on pretrained diffusion models with cross-attention layers disentangled and adopts a two-step optimization to build item-prompt associations. Versatile image editing can then be applied to specific items by manipulating the corresponding prompts. We demonstrate state-of-the-art results in four types of editing operations including image-based, text-based, mask-based editing, and item removal, covering most types of editing applications, all within a single unified framework. Notably, D-Edit is the first framework that can (1) achieve item editing through mask editing and (2) combine image and text-based editing. We demonstrate the quality and versatility of the editing results for a diverse collection of images through both qualitative and quantitative evaluations.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Negative-Prompt Inversion: Fast Image Inversion for Editing with Text-Guided Diffusion Models2025
- Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection2024
- AnyDoor: Zero-shot Object-level Image Customization2024
- An Edit Friendly DDPM Noise Space: Inversion and Manipulations2024
- DragDiffusion: Harnessing Diffusion Models for Interactive Point-Based Image Editing2024
- Blended Latent DiffusionACM Transactions on Graphics, 2023
- Emergent Correspondence from Image Diffusion2023
- What the DAAM: Interpreting Stable Diffusion Using Cross Attention2023