Unsupervised Style-based Explicit 3D Face Reconstruction from Single Image
Preprint
Abstract
Inferring 3D object structures from a single image is an ill-posed task due to depth ambiguity and occlusion. Typical resolutions in the literature include leveraging 2D or 3D ground truth for supervised learning, as well as imposing hand-crafted symmetry priors or using an implicit representation to hallucinate novel viewpoints for unsupervised methods. In this work, we propose a general adversarial learning framework for solving Unsupervised 2D to Explicit 3D Style Transfer (UE3DST). Specifically, we merge two architectures: the unsupervised explicit 3D reconstruction network of Wu et al.\ and the Generative Adversarial Network (GAN) named StarGAN-v2. We experiment across three facial datasets (Basel Face Model, 3DFAW and CelebA-HQ) and show that our solution is able to outperform well established solutions such as DepthNet in 3D reconstruction and Pix2NeRF in conditional style transfer, while we also justify the individual contributions of our model components via ablation. In contrast to the aforementioned baselines, our scheme produces features for explicit 3D rendering, which can be manipulated and utilized in downstream tasks.Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation2022
- Pix2NeRF: Unsupervised Conditional $\pi$-GAN for Single Image to Neural Radiance Fields Translation2022
- SofGAN: A Portrait Image Generator with Dynamic StylingACM Transactions on Graphics, 2022
- Warp-guided GANs for single-photo facial animationACM Transactions on Graphics, 2018
- SUN database: Large-scale scene recognition from abbey to zoo2010