Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding
Authors:
Top Cited Papers
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 11348-11358
Abstract
Image inpainting has made significant advances in recent years. However, it is still challenging to recover corrupted images with both vivid textures and reasonable structures. Some specific methods only tackle regular textures while losing holistic structures due to the limited receptive fields of convolutional neural networks (CNNs). On the other hand, attention-based models can learn better long-range dependency for the structure recovery, but they are limited by the heavy computation for inference with large image sizes. To address these issues, we propose to leverage an additional structure restorer to facilitate the image inpainting incrementally. The proposed model restores holistic image structures with a powerful attention-based transformer model in a fixed low-resolution sketch space. Such a grayscale space is easy to be upsampled to larger scales to convey correct structural information. Our structure restorer can be integrated with other pretrained inpainting models efficiently with the zero-initialized residual addition. Furthermore, a masking positional encoding strategy is utilized to improve the performance with large irregular masks. Extensive experiments on various datasets validate the efficacy of our model compared with other competitors. Our codes are released in https://github.com/DQiaole/ZITS_inpainting.Keywords
All Related Versions
Funding Information
- NSFC (62176061)
This publication has 37 references indexed in Scilit:
- Places: A 10 Million Image Database for Scene RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Image-to-Image Translation with Conditional Adversarial Networks2017
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- Perceptual Losses for Real-Time Style Transfer and Super-Resolution2016
- Microsoft COCO: Common Objects in Context2014
- Fields of Experts: A Framework for Learning Image Priors2005
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004
- Object removal by exemplar-based inpainting2003
- On the Canny edge detectorPattern Recognition, 2001
- Image inpainting2000