Region Assisted Sketch Colorization
- 31 October 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 32 (10577149) , 6142-6154
Abstract
Automatic sketch colorization is a challenging task that aims to generate a color image from a sketch, primarily due to its inherently ill-posed nature. While many approaches have shown promising results, two significant challenges remain: limited color patterns and a wide range of artifacts such as color bleeding and semantic inconsistencies among relevant regions. These issues stem from the operation of traditional convolutional structures, which capture structural features in a pixel-wise manner, resulting in inadequate utilization of regional information within the sketch. Therefore, we propose the Region-Assisted Sketch Coloring (RASC) method, which introduces an intermediate representation called the ’Region Map’ to explicitly characterize the regional information of the sketch. This Region Map is derived from the input sketch and is effectively formulated by our RASC architecture, enhancing the perception of region-wise features beyond the original pixel-wise features. Specifically, we start by employing the sketch encoder to extract hierarchical feature maps from the input sketches. Subsequently, we introduce a coarse-to-fine decoder comprising a series of Region-based Modulation (RM) blocks. This decoder modulates features that combine the modulation results of its previous block and the sketch features of the corresponding encoder block with our Region Formulation module. Each module explicitly formulates the sketch features in a region-wise manner. This accurately captures both the inner-region local style and inter-region global context dependency, resulting in various color patterns and fewer synthesis artifacts. Our experimental results show that our proposed method surpasses state-of-the-art methods in both synthetic and real sketch datasets.Keywords
Funding Information
- National Natural Science Foundation of China (NO. 61772108, NO. 61932020, NO. 61976038)
This publication has 38 references indexed in Scilit:
- Colorization Using Neural Network EnsembleIEEE Transactions on Image Processing, 2017
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Variational Exemplar-Based Image ColorizationIEEE Transactions on Image Processing, 2013
- Image colorization using similar images2012
- XDoG: An eXtended difference-of-Gaussians compendium including advanced image stylizationComputers & Graphics, 2012
- Semantic colorization with internet imagesACM Transactions on Graphics, 2011
- Vectorizing Cartoon AnimationsIEEE Transactions on Visualization and Computer Graphics, 2009
- Automatic Image Colorization Via Multimodal Predictions2008
- Variational Models for Image Colorization via Chromaticity and Brightness DecompositionIEEE Transactions on Image Processing, 2007
- Manga colorizationACM Transactions on Graphics, 2006