Joint Material and Illumination Estimation from Photo Sets in the Wild
Authors:
- Tuanfeng Y. WangTWTuanfeng Y. Wang
- Tobias RitschelTRTobias Ritschel
- Niloy J. MitraNMNiloy J. Mitra
- 1 September 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 22-31
Abstract
Faithful manipulation of shape, material, and illumination in 2D Internet images would greatly benefit from a reliable factorization of appearance into material (i.e. diffuse and specular) and illumination (i.e. environment maps). On the one hand, current methods that produce very high fidelity results, typically require controlled settings, expensive devices, or significant manual effort. To the other hand, methods that are automatic and work on 'in the wild' Internet images, often extract only low-frequency lighting or diffuse materials. In this work, we propose to make use of a set of photographs in order to jointly estimate the non-diffuse materials and sharp lighting in an uncontrolled setting. Our key observation is that seeing multiple instances of the same material under different illumination (i.e. environment), and different materials under the same illumination provide valuable constraints that can be exploited to yield a high-quality solution (i.e. specular materials and environment illumination) for all the observed materials and environments. Similar constraints also arise when observing multiple materials in a single environment, or a single material across multiple environments. Technically, we enable this by a novel scalable formulation using parametric mixture models that allows for simultaneous estimation of all materials and illumination directly from a set of (uncontrolled) Internet images. The core of this approach is an optimization procedure that uses two neural networks that are trained on synthetic images to predict good gradients in parametric space given observation of reflected light. We evaluate our method on a range of synthetic and real examples to generate high-quality estimates, qualitatively compare our results against state-of-the-art alternatives via a user study, and demonstrate photo-consistent image manipulation that is otherwise very challenging to achieve.Keywords
All Related Versions
This publication has 33 references indexed in Scilit:
- Intrinsic images in the wildACM Transactions on Graphics, 2014
- On-line learning of parametric mixture models for light transport simulationACM Transactions on Graphics, 2014
- Make it homeACM Transactions on Graphics, 2011
- A Sparse Parametric Mixture Model for BTF Compression, Editing and RenderingComputer Graphics Forum, 2011
- Goal‐based CausticsComputer Graphics Forum, 2011
- All-frequency precomputed radiance transfer using spherical radial basis functions and clustered tensor approximationACM Transactions on Graphics, 2006
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004
- A data-driven reflectance modelACM Transactions on Graphics, 2003
- Algorithm 778: L-BFGS-BACM Transactions on Mathematical Software, 1997
- Directional Reflectance and Emissivity of an Opaque SurfaceApplied Optics, 1965