Total relighting
Authors:
Open Access
- 19 July 2021
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 40 (4) , 1-21
Abstract
We propose a novel system for portrait relighting and background replacement, which maintains high-frequency boundary details and accurately synthesizes the subject's appearance as lit by novel illumination, thereby producing realistic composite images for any desired scene. Our technique includes foreground estimation via alpha matting, relighting, and compositing. We demonstrate that each of these stages can be tackled in a sequential pipeline without the use of priors (e.g. known background or known illumination) and with no specialized acquisition techniques, using only a single RGB portrait image and a novel, target HDR lighting environment as inputs. We train our model using relit portraits of subjects captured in a light stage computational illumination system, which records multiple lighting conditions, high quality geometry, and accurate alpha mattes. To perform realistic relighting for compositing, we introduce a novel per-pixel lighting representation in a deep learning framework, which explicitly models the diffuse and the specular components of appearance, producing relit portraits with convincingly rendered non-Lambertian effects like specular highlights. Multiple experiments and comparisons show the effectiveness of the proposed approach when applied to in-the-wild images.Keywords
This publication has 39 references indexed in Scilit:
- Image based relighting using neural networksACM Transactions on Graphics, 2015
- Shape, Illumination, and Reflectance from ShadingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2014
- Circularly polarized spherical illumination reflectometryACM Transactions on Graphics, 2010
- Performance relighting and reflectance transformation with time-multiplexed illumination2005
- Reflectance field rendering of human faces for "Spider-Man 2"2005
- A lighting reproduction approach to live-action compositingACM Transactions on Graphics, 2002
- Color transfer between imagesIEEE Computer Graphics and Applications, 2001
- Acquiring the reflectance field of a human face2000
- Interactive design of complex time dependent lightingIEEE Computer Graphics and Applications, 1995
- Illumination for computer generated picturesCommunications of the ACM, 1975