Learning Illumination from Diverse Portraits
Authors:
Open Access
- 17 November 2020
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-4
Abstract
We present a learning-based technique for estimating high dynamic range (HDR), omnidirectional illumination from a single low dynamic range (LDR) portrait image captured under arbitrary indoor or outdoor lighting conditions. We train our model using portrait photos paired with their ground truth illumination. We generate a rich set of such photos by using a light stage to record the reflectance field and alpha matte of 70 diverse subjects in various expressions. We then relight the subjects using image-based relighting with a database of one million HDR lighting environments, compositing them onto paired high-resolution background imagery recorded during the lighting acquisition. We train the lighting estimation model using rendering-based loss functions and add a multi-scale adversarial loss to estimate plausible high frequency lighting detail. We show that our technique outperforms the state-of-the-art technique for portrait-based lighting estimation, and we also show that our method reliably handles the inherent ambiguity between overall lighting strength and surface albedo, recovering a similar scale of illumination for subjects with diverse skin tones. Our method allows virtual objects and digital characters to be added to a portrait photograph with consistent illumination. As our inference runs in real-time on a smartphone, we enable realistic rendering and compositing of virtual objects into live video for augmented reality.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Faces as Lighting Probes via Unsupervised Deep Highlight Extraction2018
- SfSNet: Learning Shape, Reflectance and Illuminance of Faces 'in the Wild'2018
- Occlusion-Aware 3D Morphable Models and an Illumination Prior for Face Image AnalysisInternational Journal of Computer Vision, 2018
- Learning to predict indoor illumination from a single imageACM Transactions on Graphics, 2017
- Deep outdoor illumination estimation2017
- Faces as light probes for relightingOptical Engineering, 2012
- A lighting reproduction approach to live-action compositingACM Transactions on Graphics, 2002
- On the relationship between radiance and irradiance: determining the illumination from images of a convex Lambertian objectJournal of the Optical Society of America A, 2001
- Acquiring the reflectance field of a human face2000
- Rendering synthetic objects into real scenes1998