RePaint: Inpainting using Denoising Diffusion Probabilistic Models
Authors:
- Andres RomeroARAndres Romero
- Fisher YuFYFisher Yu
- Radu TimofteRTRadu Timofte
Preprint
Abstract
Free-form inpainting is the task of adding new content to an image in the regions specified by an arbitrary binary mask. Most existing approaches train for a certain distribution of masks, which limits their generalization capabilities to unseen mask types. Furthermore, training with pixel-wise and perceptual losses often leads to simple textural extensions towards the missing areas instead of semantically meaningful generation. In this work, we propose RePaint: A Denoising Diffusion Probabilistic Model (DDPM) based inpainting approach that is applicable to even extreme masks. We employ a pretrained unconditional DDPM as the generative prior. To condition the generation process, we only alter the reverse diffusion iterations by sampling the unmasked regions using the given image information. Since this technique does not modify or condition the original DDPM network itself, the model produces high-quality and diverse output images for any inpainting form. We validate our method for both faces and general-purpose image inpainting using standard and extreme masks. RePaint outperforms state-of-the-art Autoregressive, and GAN approaches for at least five out of six mask distributions. Github Repository: git.io/RePaintKeywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Aggregated Contextual Transformations for High-Resolution Image InpaintingIEEE Transactions on Visualization and Computer Graphics, 2022
- High-Fidelity Pluralistic Image Completion with Transformers2021
- ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models2021
- Denoising Synthesis: A module for fast image synthesis using denoising-based modelsSoftware Impacts, 2021
- NTIRE 2021 Learning the Super-Resolution Space Challenge2021
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- Generative Image Inpainting with Contextual Attention2018
- Places: A 10 Million Image Database for Scene RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Simultaneous structure and texture image inpaintingIEEE Transactions on Image Processing, 2003
- Filling-in by joint interpolation of vector fields and gray levelsIEEE Transactions on Image Processing, 2001