Double Reference Guided Interactive 2D and 3D Caricature Generation
Open Access
- 16 December 2024
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 21 (1) , 1-21
Abstract
In this article, we propose the first geometry and texture (double) referenced interactive two-dimensional (2D) and 3D caricature generating and editing method. The main challenge of caricature generation lies in the fact that it not only exaggerates the facial geometry but also refreshes the facial texture. We address this challenge by utilizing the semantic segmentation maps as an intermediary domain, removing the influence of photo texture while preserving the person-specific geometry features. Specifically, our proposed method consists of two main components: 3D-CariNet and CariMaskGAN. 3D-CariNet uses sketches or caricatures to exaggerate the input photo into several types of 3D caricatures. To generate a CariMask, we geometrically exaggerate the photos using the projection of exaggerated 3D landmarks, after which CariMask is converted into a caricature by CariMaskGAN. In this step, users can edit and adjust the geometry of caricatures freely. Moreover, we propose a semantic detail preprocessing approach that considerably increases the details of generated caricatures and allows modification of hair strands, wrinkles, and beards. By rendering high-quality 2D caricatures as textures, we produce 3D caricatures with a variety of texture styles. Extensive experimental results have demonstrated that our method can produce higher-quality caricatures as well as support interactive modification with ease.Keywords
Funding Information
- National Natural Science Foundation of China (6207071897)
- National Natural Science Regional Joint Foundation of China (U19A2063)
This publication has 40 references indexed in Scilit:
- Learning to Caricature via Semantic Shape TransformInternational Journal of Computer Vision, 2021
- Landmark Detection and 3D Face Reconstruction for Caricature using a Nonlinear Parametric ModelGraphical Models, 2021
- CariGAN: Caricature generation through weakly paired adversarial learningNeural Networks, 2020
- Unpaired photo-to-caricature translation on faces in the wildNeurocomputing, 2019
- Multimodal Unsupervised Image-to-Image Translation2018
- BiSeNet: Bilateral Segmentation Network for Real-Time Semantic Segmentation2018
- SphereFace: Deep Hypersphere Embedding for Face Recognition2017
- Deep Residual Learning for Image Recognition2016
- FaceWarehouse: A 3D Facial Expression Database for Visual ComputingIEEE Transactions on Visualization and Computer Graphics, 2013
- Novel approach to neural network based caricature generation2006