Self-Prior Guided Pixel Adversarial Networks for Blind Image Inpainting
- 9 June 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 45 (10) , 12377-12393
Abstract
Blind image inpainting involves two critical aspects, i.e. , “where to inpaint” and “how to inpaint”. Knowing “where to inpaint” can eliminate the interference arising from corrupted pixel values; a good “how to inpaint” strategy yields high-quality inpainted results robust to various corruptions. In existing methods, these two aspects usually lack explicit and separate consideration. This paper fully explores these two aspects and proposes a self-prior guided inpainting network (SIN). The self-priors are obtained by detecting semantic-discontinuous regions and by predicting global semantic structures of the input image. On the one hand, the self-priors are incorporated into the SIN, which enables the SIN to perceive valid context information from uncorrupted regions and to synthesize semantic-aware textures for corrupted regions. On the other hand, the self-priors are reformulated to provide a pixel-wise adversarial feedback and a high-level semantic structure feedback, which can promote the semantic continuity of inpainted images. Experimental results demonstrate that our method achieves state-of-the-art performance in metric scores and in visual quality. It has an advantage over many existing methods that assume “where to inpaint” is known in advance. Extensive experiments on a series of related image restoration tasks validate the effectiveness of our method in obtaining high-quality inpainting.Keywords
Funding Information
- National Key Research and Development Program of China (2020AAA0106800)
- National Natural Science Foundation of China (62202470, 61972397, 62122086, U1936204, 62036011, 62192782, 61721004, U2033210)
- Beijing Natural Science Foundation (4224093, JQ21017, L223003)
- Major Projects of Guangdong Education Department for Foundation Research and Applied Research (2017KZDXM081, 2018KZDXM066)
- Guangdong Provincial University Innovation Team Project (2020KCXTD045)
- Youth Innovation Promotion Association of the Chinese Academy of Sciences
This publication has 45 references indexed in Scilit:
- Context Encoders: Feature Learning by Inpainting2016
- Deep Residual Learning for Image Recognition2016
- The Cityscapes Dataset for Semantic Urban Scene Understanding2016
- Rethinking the Inception Architecture for Computer Vision2016
- Blind inpainting using the fully convolutional neural networkThe Visual Computer, 2015
- Deep Learning Face Attributes in the Wild2015
- The Pascal Visual Object Classes Challenge: A RetrospectiveInternational Journal of Computer Vision, 2014
- Image meldingACM Transactions on Graphics, 2012
- Space-Time Completion of VideoIEEE Transactions on Pattern Analysis and Machine Intelligence, 2007
- Digital inpainting based on the Mumford–Shah–Euler image modelEuropean Journal of Applied Mathematics, 2002