LIME: Live Intrinsic Material Estimation
Authors:
- Abhimitra MekaAMAbhimitra Meka
- Maxim MaximovMMMaxim Maximov
- Michael ZollhoferMZMichael Zollhofer
- Avishek ChatterjeeACAvishek Chatterjee
- Hans-Peter SeidelHSHans-Peter Seidel
- Christian RichardtCRChristian Richardt
- Christian TheobaltCTChristian Theobalt
- 1 June 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 6315-6324
Abstract
We present the first end-to-end approach for real-time material estimation for general object shapes with uniform material that only requires a single color image as input. In addition to Lambertian surface properties, our approach fully automatically computes the specular albedo, material shininess, and a foreground segmentation. We tackle this challenging and ill-posed inverse rendering problem using recent advances in image-to-image translation techniques based on deep convolutional encoder-decoder architectures. The underlying core representations of our approach are specular shading, diffuse shading and mirror images, which allow to learn the effective and accurate separation of diffuse and specular albedo. In addition, we propose a novel highly efficient perceptual rendering loss that mimics real-world image formation and obtains intermediate results even during run time. The estimation of material parameters at real-time frame rates enables exciting mixed-reality applications, such as seamless illumination-consistent integration of virtual objects into real-world scenes, and virtual material cloning. We demonstrate our approach in a live setup, compare it to the state of the art, and demonstrate its effectiveness through quantitative and qualitative evaluation.Keywords
All Related Versions
This publication has 29 references indexed in Scilit:
- Reflectance and Illumination Recovery in the WildIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- AppFusion: Interactive Appearance Acquisition Using a Kinect SensorComputer Graphics Forum, 2015
- AppImACM Transactions on Graphics, 2014
- Multiple Light Source Estimation in a Single ImageComputer Graphics Forum, 2013
- Estimating the Natural Illumination Conditions from a Single Outdoor ImageInternational Journal of Computer Vision, 2011
- KinectFusion2011
- Illumination decomposition for material recoloring with consistent interreflectionsACM Transactions on Graphics, 2011
- Image-based material editing2006
- A signal-processing framework for inverse rendering2001
- The rendering equationACM SIGGRAPH Computer Graphics, 1986