Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing
Preprint
Abstract
Despite recent advances in large-scale text-to-image generative models, manipulating real images with these models remains a challenging problem. The main limitations of existing editing methods are that they either fail to perform with consistent quality on a wide range of image edits or require time-consuming hyperparameter tuning or fine-tuning of the diffusion model to preserve the image-specific appearance of the input image. We propose a novel approach that is built upon a modified diffusion sampling process via the guidance mechanism. In this work, we explore the self-guidance technique to preserve the overall structure of the input image and its local regions appearance that should not be edited. In particular, we explicitly introduce layout-preserving energy functions that are aimed to save local and global structures of the source image. Additionally, we propose a noise rescaling mechanism that allows to preserve noise distribution by balancing the norms of classifier-free guidance and our proposed guiders during generation. Such a guiding approach does not require fine-tuning the diffusion model and exact inversion process. As a result, the proposed method provides a fast and high-quality editing mechanism. In our experiments, we show through human evaluation and quantitative analysis that the proposed method allows to produce desired editing which is more preferable by humans and also achieves a better trade-off between editing quality and preservation of the original image. Our code is available at https://github.com/FusionBrainLab/Guide-and-Rescale.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Negative-Prompt Inversion: Fast Image Inversion for Editing with Text-Guided Diffusion Models2025
- ProxEdit: Improving Tuning-Free Real Image Editing with Proximal Guidance2024
- Effective Real Image Editing with Accelerated Iterative Diffusion Inversion2023
- Zero-shot Image-to-Image Translation2023
- Null-text Inversion for Editing Real Images using Guided Diffusion Models2023
- InstructPix2Pix: Learning to Follow Image Editing Instructions2023
- EDICT: Exact Diffusion Inversion via Coupled Transformations2023
- SINE: SINgle Image Editing with Text-to-Image Diffusion Models2023