CLIP-Guided StyleGAN Inversion for Text-Driven Real Image Editing
Preprint
Abstract
Researchers have recently begun exploring the use of StyleGAN-based models for real image editing. One particularly interesting application is using natural language descriptions to guide the editing process. Existing approaches for editing images using language either resort to instance-level latent code optimization or map predefined text prompts to some editing directions in the latent space. However, these approaches have inherent limitations. The former is not very efficient, while the latter often struggles to effectively handle multi-attribute changes. To address these weaknesses, we present CLIPInverter, a new text-driven image editing approach that is able to efficiently and reliably perform multi-attribute changes. The core of our method is the use of novel, lightweight text-conditioned adapter layers integrated into pretrained GAN-inversion networks. We demonstrate that by conditioning the initial inversion step on the CLIP embedding of the target description, we are able to obtain more successful edit directions. Additionally, we use a CLIP-guided refinement step to make corrections in the resulting residual latent codes, which further improves the alignment with the text prompt. Our method outperforms competing approaches in terms of manipulation accuracy and photo-realism on various domains including human faces, cats, and birds, as shown by our qualitative and quantitative results.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- Imagic: Text-Based Real Image Editing with Diffusion Models2023
- Style Transformer for Image Inversion and Editing2022
- VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks2022
- StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation2021
- Image2StyleGAN++: How to Edit the Embedded Images?2020
- Editing in Style: Uncovering the Local Semantics of GANs2020
- ArcFace: Additive Angular Margin Loss for Deep Face Recognition2019
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- Efficient Parametrization of Multi-domain Deep Neural Networks2018
- Deep Residual Learning for Image Recognition2016