Blind Inpainting of Large-scale Masks of Thin Structures with Adversarial and Reinforcement Learning
Preprint
Abstract
Several imaging applications (vessels, retina, plant roots, road networks from satellites) require the accurate segmentation of thin structures for subsequent analysis. Discontinuities (gaps) in the extracted foreground may hinder down-stream image-based analysis of biomarkers, organ structure and topology. In this paper, we propose a general post-processing technique to recover such gaps in large-scale segmentation masks. We cast this problem as a blind inpainting task, where the regions of missing lines in the segmentation masks are not known to the algorithm, which we solve with an adversarially trained neural network. One challenge of using large images is the memory capacity of current GPUs. The typical approach of dividing a large image into smaller patches to train the network does not guarantee global coherence of the reconstructed image that preserves structure and topology. We use adversarial training and reinforcement learning (Policy Gradient) to endow the model with both global context and local details. We evaluate our method in several datasets in medical imaging, plant science, and remote sensing. Our experiments demonstrate that our model produces the most realistic and complete inpainted results, outperforming other approaches. In a dedicated study on plant roots we find that our approach is also comparable to human performance. Implementation available at \url{https://github.com/Hhhhhhhhhhao/Thin-Structure-Inpainting}.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Foreground-Aware Image Inpainting2019
- Image Inpainting for Irregular Holes Using Partial Convolutions2018
- Learning to restore deteriorated line drawingThe Visual Computer, 2018
- A Deep Residual convolutional neural network for facial keypoint detection with missing labelsSignal Processing, 2018
- Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks2017
- Joint Gap Detection and Inpainting of Line Drawings2017
- Image-to-Image Translation with Conditional Adversarial Networks2017
- Context Encoders: Feature Learning by Inpainting2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Retinal vessel segmentation by improved matched filtering: evaluation on a new high‐resolution fundus image databaseIET Image Processing, 2013