Real-time shading-based refinement for consumer depth cameras
Open Access
- 19 November 2014
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 33 (6) , 1-10
Abstract
We present the first real-time method for refinement of depth data using shape-from-shading in general uncontrolled scenes. Per frame, our real-time algorithm takes raw noisy depth data and an aligned RGB image as input, and approximates the time-varying incident lighting, which is then used for geometry refinement. This leads to dramatically enhanced depth maps at 30Hz. Our algorithm makes few scene assumptions, handling arbitrary scene objects even under motion. To enable this type of real-time depth map enhancement, we contribute a new highly parallel algorithm that reformulates the inverse rendering optimization problem in prior work, allowing us to estimate lighting and shape in a temporally coherent way at video frame-rates. Our optimization problem is minimized using a new regular grid Gauss-Newton solver implemented fully on the GPU. We demonstrate results showing enhanced depth maps, which are comparable to offline methods but are computed orders of magnitude faster, as well as baseline comparisons with online filtering-based methods. We conclude with applications of our higher quality depth maps for improved real-time surface reconstruction and performance capture.Keywords
Funding Information
- Max Planck Center for Visual Computing and Communication
- Deutsche Forschungsgemeinschaft (GRK-1773 Heterogeneous Image Systems)
- European Research Council (335545 CapReal)
This publication has 37 references indexed in Scilit:
- Acquiring reflectance and shape from continuous spherical harmonic illuminationACM Transactions on Graphics, 2013
- Lightweight binocular facial performance capture under uncontrolled lightingACM Transactions on Graphics, 2012
- Efficient GPU Data Structures and Methods to Solve Sparse Linear Systems in Dynamics ApplicationsComputer Graphics Forum, 2012
- Coherent Spatiotemporal Filtering, Upsampling and Rendering of RGBZ VideosComputer Graphics Forum, 2012
- Multiview face capture using polarized spherical gradient illuminationACM Transactions on Graphics, 2011
- Fusion of stereo vision and Time-Of-Flight imaging for improved 3D estimationInternational Journal of Intelligent Systems Technologies and Applications, 2008
- Joint bilateral upsamplingACM Transactions on Graphics, 2007
- Shape-from-shading: a surveyIEEE Transactions on Pattern Analysis and Machine Intelligence, 1999
- Learning shape from shading by a multilayer networkIEEE Transactions on Neural Networks, 1996
- Determining lightness from an imageComputer Graphics and Image Processing, 1974