Sketch2Human: Deep Human Generation With Disentangled Geometry and Appearance Constraints
- 23 May 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 31 (9) , 4480-4492
Abstract
Geometry- and appearance-controlled full-body human image generation is an interesting but challenging task. Existing solutions are either unconditional or dependent on coarse conditions (e.g., pose, text), thus lacking explicit geometry and appearance control of body and garment. Sketching offers such editing ability and has been adopted in various sketch-based face generation and editing solutions. However, directly adapting sketch-based face generation to full-body generation often fails to produce high-fidelity and diverse results due to the high complexity and diversity in the pose, body shape, and garment shape and texture. Recent geometrically controllable diffusion-based methods mainly rely on prompts to generate appearance. It is hard to balance the realism and the faithfulness of their results to the sketch when the input is coarse. This work presents Sketch2Human, the first system for controllable full-body human image generation guided by a semantic sketch (for geometry control) and a reference image (for appearance control). Our solution is based on the latent space of StyleGAN-Human with inverted geometry and appearance latent codes as input. Specifically, we present a sketch encoder trained with a large synthetic dataset sampled from StyleGAN-Human's latent space and directly supervised by sketches rather than real images. Considering the entangled information of partial geometry and texture in StyleGAN-Human and the absence of disentangled datasets, we design a novel training scheme that creates geometry-preserved and appearance-transferred training data to tune a generator to achieve disentangled geometry and appearance control. Although our method is trained with synthetic data, it can also handle hand-drawn sketches. Qualitative and quantitative evaluations demonstrate the superior performance of our method to state-of-the-art methods.Keywords
All Related Versions
Funding Information
- Chow Sang Sang Group (9229119)
- Research Grants Council of the Hong Kong Special Administrative Region, China (CityU PDFS2223-1S10)
- Guangdong Provincial Outstanding Youth (2023B1515020055)
- NSFC (62172348, 61931024)
- Shenzhen General (JCYJ20220530143604010)
This publication has 47 references indexed in Scilit:
- DeepPortraitDrawing: Generating human body images from freehand sketchesComputers & Graphics, 2023
- DynaST: Dynamic Sparse Transformer for Exemplar-Guided Image Generation2022
- StyleGAN-Human: A Data-Centric Odyssey of Human Generation2022
- HumanGAN: A Generative Model of Human Images2021
- Abstract: nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation2019
- High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs2018
- VITON: An Image-Based Virtual Try-on Network2018
- Deformable GANs for Pose-Based Human Image Generation2018
- SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis2018
- DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations2016