NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo
Authors:
Top Cited Papers
- 1 October 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 5590-5599
Abstract
In this work, we present a new multi-view depth estimation method that utilizes both conventional SfM reconstruction and learning-based priors over the recently proposed neural radiance fields (NeRF). Unlike existing neural network based optimization method that relies on estimated correspondences, our method directly optimizes over implicit volumes, eliminating the challenging step of matching pixels in indoor scenes. The key to our approach is to utilize the learning-based priors to guide the optimization process of NeRF. Our system firstly adapts a monocular depth network over the target scene by finetuning on its sparse SfM reconstruction. Then, we show that the shape-radiance ambiguity of NeRF still exists in indoor environments and propose to address the issue by employing the adapted depth priors to monitor the sampling process of volume rendering. Finally, a per-pixel confidence map acquired by error computation on the rendered image can be used to fur ther improve the depth quality. Experiments show that our proposed framework significantly outperforms state-of-the-art methods on indoor scenes, with surprising findings presented on the effectiveness of correspondence-based opti-mization and NeRF-based optimization over the adapted depth priors. In addition, we show that the guided opti-mization scheme does not sacrifice the original synthesis capability of neural radiance fields, improving the rendering quality on both seen and novel views. Code is available at https://github.com/weiyithu/NerfingMVS.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China
This publication has 42 references indexed in Scilit:
- ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes2017
- Structure-from-Motion Revisited2016
- Fast Cost-Volume Filtering for Visual Correspondence and BeyondIEEE Transactions on Pattern Analysis and Machine Intelligence, 2012
- PatchMatch Stereo - Stereo Matching with Slanted Support Windows2011
- Real-Time Plane-Sweeping Stereo with Multiple Sweeping Directions2007
- Adaptive support-weight approach for correspondence searchIEEE Transactions on Pattern Analysis and Machine Intelligence, 2006
- Multi-View Stereo via Volumetric Graph-Cuts2005
- A Theory of Shape by Space CarvingInternational Journal of Computer Vision, 2000
- The lumigraph1996
- Light field rendering1996