From Sketch to Reality: Enabling High-Quality, Cross-Category 3D Model Generation From Free-Hand Sketches With Minimal Data
- 1 July 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 32 (7) , 5088-5097
Abstract
This paper presents a novel approach for generating high-quality, cross-category 3D models from free-hand sketches with limited training data. We propose the first semi-supervised learning method to our knowledge for sketch-to-3D model conversion. Innovatively, we design a coarse-to-fine pipeline to perform the semi-supervised learning in the coarse stage and train a diffusion-based refiner to get a high-resolution 3D model. We designed a sketch-augmentation method for semi-supervised learning and integrated priors such as CLIP loss, shape prototypes, and adversarial loss to help generate high-quality results even with abstract and imprecise sketches. We also introduce an innovative procedural 3D generation method based on CAD code, which helps pre-train part of the network before fine-tuning with limited real data. Our approach, coupled with a specifically designed curriculum learning, allows us to generate high-quality 3D models across multiple categories with as few as 300 sketch-3D model pairs, marking a significant advancement over previous single-category approaches. In addition, we introduce the KO2D dataset, the largest collection of hand-drawn sketch-3D pairs to support further research in this area. As sketches are a far more intuitive and detailed way for users to express their unique ideas, we believe that this paper can move us closer to democratizing 3D content creation, enabling anyone to transform their ideas into 3D models effortlessly.Keywords
Funding Information
- National Natural Science Foundation of China (625B2169)
- ZJU Kunpeng & Ascend Center of Excellence
- Dream Set Off - Kunpeng & Ascend Seed Program
This publication has 44 references indexed in Scilit:
- Deep3DSketch+: Rapid 3D Modeling from Single Free-Hand Sketches2023
- Sketch-specific data augmentation for freehand sketch recognitionNeurocomputing, 2021
- Sketch2Model: View-Aware 3D Modeling from Single Free-Hand Sketches2021
- Interactive modeling of lofted shapes from a single imageComputational Visual Media, 2019
- Sketch-based interaction and modeling: where do we stand?Artificial Intelligence for Engineering Design, Analysis and Manufacturing, 2019
- Development of knowledge based parametric CAD modeling system for spur gear: An approachAlexandria Engineering Journal, 2018
- Learning Single-View 3D Reconstruction with Limited Pose Supervision2018
- Deep Co-Training for Semi-Supervised Image Recognition2018
- Sketch-based modeling: A surveyComputers & Graphics, 2009
- Teaching for CAD expertiseInternational Journal of Technology and Design Education, 2007