CustomSketching: Sketch Concept Extraction for Sketch‐based Image Synthesis and Editing
Open Access
- 1 October 2024
- journal article
- research article
- Published by Wiley in Computer Graphics Forum
- Vol. 43 (7)
Abstract
Personalization techniques for large text‐to‐image (T2I) models allow users to incorporate new concepts from reference images. However, existing methods primarily rely on textual descriptions, leading to limited control over customized images and failing to support fine‐grained and local editing (e.g., shape, pose, and details). In this paper, we identify sketches as an intuitive and versatile representation that can facilitate such control, e.g., contour lines capturing shape information and flow lines representing texture. This motivates us to explore a novel task of sketch concept extraction: given one or more sketch‐image pairs, we aim to extract a special sketch concept that bridges the correspondence between the images and sketches, thus enabling sketch‐based image synthesis and editing at a fine‐grained level. To accomplish this, we introduce CustomSketching, a two‐stage framework for extracting novel sketch concepts via few‐shot learning. Considering that an object can often be depicted by a contour for general shapes and additional strokes for internal details, we introduce a dual‐sketch representation to reduce the inherent ambiguity in sketch depiction. We employ a shape loss and a regularization loss to balance fidelity and editability during optimization. Through extensive experiments, a user study, and several applications, we show our method is effective and superior to the adapted baselines.Keywords
All Related Versions
Funding Information
- Hong Kong University of Science and Technology
This publication has 39 references indexed in Scilit:
- Multi-Concept Customization of Text-to-Image Diffusion2023
- Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation2023
- SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches2022
- Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation2021
- TediGAN: Text-Guided Diverse Face Image Generation and Manipulation2021
- Cross-Modal Contrastive Learning for Text-to-Image Generation2021
- A Style-Based Generator Architecture for Generative Adversarial Networks2019
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks2018
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis2018