Free-view Face Relighting Using a Hybrid Parametric Neural Model on a SMALL-OLAT Dataset
Open Access
- 7 January 2023
- journal article
- research article
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 131 (4) , 1002-1021
Abstract
The development of neural relighting techniques has by far outpaced the rate of their corresponding training data (e.g., OLAT) generation. For example, high-quality relighting from a single portrait image still requires supervision from comprehensive datasets covering broad diversities in gender, race, complexion, and facial geometry. We present a hybrid parametric neural relighting (PN-Relighting) framework for single portrait relighting, using a much smaller OLAT dataset or SMOLAT. At the core of PN-Relighting, we employ parametric 3D faces coupled with appearance inference and implicit material modelling to enrich SMOLAT for handling in-the-wild images. Specifically, we tailor an appearance inference module to generate detailed geometry and albedo on top of the parametric face and develop a neural rendering module to first construct an implicit material representation from SMOLAT and then conduct self-supervised training on in-the-wild image datasets. Comprehensive experiments show that PN-Relighting produces comparable high-quality relighting to TotalRelighting (Pandey et al., 2021), but with a smaller dataset. It further improves shape estimation and naturally supports free-viewpoint rendering and partial skin material editing. PN-Relighting also serves as a data augmenter to produce rich OLAT datasets beyond the original capture.Keywords
This publication has 67 references indexed in Scilit:
- Dynamic 3D avatar creation from hand-held video inputACM Transactions on Graphics, 2015
- Dense 3D face alignment from 2D videos in real-time2015
- Automatic acquisition of high-fidelity facial performances using monocular videosACM Transactions on Graphics, 2014
- Real-time 3D face tracking based on active appearance model constrained by depth dataImage and Vision Computing, 2014
- Realtime facial animation with on-the-fly correctivesACM Transactions on Graphics, 2013
- High-quality single-shot capture of facial geometryACM Transactions on Graphics, 2010
- Reflectance field rendering of human faces for "Spider-Man 2"2005
- Lambertian reflectance and linear subspacesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2003
- Photometric Method For Determining Surface Orientation From Multiple ImagesOptical Engineering, 1980
- Illumination for computer generated picturesCommunications of the ACM, 1975