Learning Unsupervised Multi-View Stereopsis via Robust Photometric Consistency
Preprint
Abstract
We present a learning based approach for multi-view stereopsis (MVS). While current deep MVS methods achieve impressive results, they crucially rely on ground-truth 3D training data, and acquisition of such precise 3D geometry for supervision is a major hurdle. Our framework instead leverages photometric consistency between multiple views as supervisory signal for learning depth prediction in a wide baseline MVS setup. However, naively applying photo consistency constraints is undesirable due to occlusion and lighting changes across views. To overcome this, we propose a robust loss formulation that: a) enforces first order consistency and b) for each point, selectively enforces consistency with some views, thus implicitly handling occlusions. We demonstrate our ability to learn MVS without 3D supervision using a real dataset, and show that each component of our proposed robust loss results in a significant improvement. We qualitatively observe that our reconstructions are often more complete than the acquired ground truth, further showing the merits of this approach. Lastly, our learned model generalizes to novel settings, and our approach allows adaptation of existing CNNs to datasets without ground-truth 3D by unsupervised finetuning. Project webpage: https://tejaskhot.github.io/unsup_mvsKeywords
All Related Versions
This publication has 10 references indexed in Scilit:
- ActiveStereoNet: End-to-End Self-supervised Learning for Active Stereo Systems2018
- End-to-End Learning of Geometry and Context for Deep Stereo Regression2017
- Unsupervised Monocular Depth Estimation with Left-Right Consistency2017
- Deep3D: Fully Automatic 2D-to-3D Video Conversion with Deep Convolutional Neural Networks2016
- Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue2016
- Large-Scale Data for Multiple-View StereopsisInternational Journal of Computer Vision, 2016
- Multi-View Stereo: A TutorialFoundations and Trends in Computer Graphics and Vision, 2015
- Accurate Multiple View 3D Reconstruction Using Patch-Based Stereo for Large-Scale ScenesIEEE Transactions on Image Processing, 2013
- Accurate, Dense, and Robust Multiview StereopsisIEEE Transactions on Pattern Analysis and Machine Intelligence, 2009
- Multi-View Stereo for Community Photo Collections2007