ENGDM: Enhanced Non-Isotropic Gaussian Diffusion Model for Progressive Image Editing
Open Access
Abstract
Diffusion models have made remarkable progress in image generation, leading to advancements in the field of image editing. However, balancing editability with faithfulness remains a significant challenge. Motivated by the fact that more novel content will be generated when larger variance noise is applied to the image, in this paper, we propose an Enhanced Non-isotropic Gaussian Diffusion Model (ENGDM) for progressive image editing, which introduces independent Gaussian noise with varying variances to each pixel based on its editing needs. To enable efficient inference without retraining, ENGDM is rectified into an isotropic Gaussian diffusion model (IGDM) by assigning different total diffusion times to different pixels. Furthermore, we introduce reinforced text embeddings, using a novel editing reinforcement loss in the latent space to optimize text embeddings for enhanced editability. And we introduce optimized noise variances by employing a structural consistency loss to dynamically adjust the denoising time steps for each pixel for better faithfulness. Experimental results on multiple datasets demonstrate that ENGDM achieves state-of-the-art performance in image-editing tasks, effectively balancing faithfulness to the source image and alignment with the desired editing target.Keywords
Funding Information
- Key-Area Research and Development Program of Guangdong Province (2022B0303020003, 12426313, 12326615, 623B2084, BX20240276, xzy022025047)
- NSFC (2022B0303020003, 12426313, 12326615, 623B2084, BX20240276, xzy022025047)
- China National Postdoctotal Program for Innovative Talents (2022B0303020003, 12426313, 12326615, 623B2084, BX20240276, xzy022025047)
- Fundamental Research Funds for the Central Universities, China (2022B0303020003, 12426313, 12326615, 623B2084, BX20240276, xzy022025047)
This publication has 26 references indexed in Scilit:
- PFB-Diff: Progressive Feature Blending diffusion for text-driven image editingNeural Networks, 2024
- ZONE: Zero-Shot Instruction-Guided Local Editing2024
- Blended Latent DiffusionACM Transactions on Graphics, 2023
- Null-text Inversion for Editing Real Images using Guided Diffusion Models2023
- InstructPix2Pix: Learning to Follow Image Editing Instructions2023
- EDICT: Exact Diffusion Inversion via Coupled Transformations2023
- Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation2023
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Blended Diffusion for Text-driven Editing of Natural Images2022
- A Connection Between Score Matching and Denoising AutoencodersNeural Computation, 2011