MaskSketch: Unpaired Structure-guided Masked Image Generation
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1879-1889
Abstract
Recent conditional image generation methods produce images of remarkable diversity, fidelity and realism. However, the majority of these methods allow conditioning only on labels or text prompts, which limits their level of control over the generation result. In this paper, we introduce MaskSketch, an image generation method that allows spatial conditioning of the generation result using a guiding sketch as an extra conditioning signal during sampling. MaskSketch utilizes a pretrained masked generative transformer, requiring no model training or paired supervision, and works with input sketches of different levels of abstraction. We show that intermediate self-attention maps of a masked generative transformer encode important structural information of the input image, such as scene layout and object shape, and we propose a novel sampling method based on this observation to enable structure-guided generation. Our results show that MaskSketch achieves high image realism and fidelity to the guiding structure. Evaluated on standard benchmark datasets, MaskSketch outperforms state-of-the-art methods for sketch-to-image translation, as well as unpaired image-to-image translation approaches. The code can be found on our project website: https://masksketch.github.io/Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Mask-Predict: Parallel Decoding of Conditional Masked Language Models2019
- Image Generation from Sketch Constraint Using Contextual GAN2018
- StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation2018
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis2018
- Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks2017
- Neural Information Processing2017
- Holistically-Nested Edge Detection2015
- Photosketcher: Interactive Sketch-Based Image SynthesisIEEE Computer Graphics and Applications, 2011
- ImageNet: A large-scale hierarchical image database2009