MorpheuS: Neural Dynamic $360^{\circ}$ Surface Reconstruction from Monocular RGB-D Video
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 20965-20976
Abstract
Neural rendering has demonstrated remarkable success in dynamic scene reconstruction. Thanks to the expressiveness of neural representations, prior works can accurately capture the motion and achieve high-fidelity reconstruction of the target object. Despite this, real-world video sce-narios often feature large unobserved regions where neural representations struggle to achieve realistic completion. To tackle this challenge, we introduce MorpheuS, a framework for dynamic $360^\circ$ surface reconstruction from a casually captured RGB-D video. Our approach models the target scene as a canonical field that encodes its geometry and appearance, in conjunction with a defor-mation field that warps points from the current frame to the canonical space. We leverage a view-dependent diffusion prior and distill knowledge from it to achieve realistic completion of unobserved regions. Experimental results on various real-world and synthetic datasets show that our method can achieve high-fidelity 360° surface reconstruction of a deformable object from a monocular RGB-D video. Project page: https: / /hengyi wang. gi thub. io/ pro jects/morpheus.
Keywords
All Related Versions
This publication has 67 references indexed in Scilit:
- GO-Surf: Neural Feature Grid Optimization for Fast, High-Fidelity RGB-D Surface Reconstruction2022
- Learning Category-Specific Mesh Reconstruction from Image Collections2018
- KillingFusion: Non-rigid 3D Reconstruction without Correspondences2017
- 3D Menagerie: Modeling the 3D Shape and Pose of Animals2017
- VolumeDeform: Real-Time Volumetric Non-rigid Reconstruction2016
- Face2Face: Real-Time Face Capture and Reenactment of RGB Videos2016
- Total Moving Face Reconstruction2014
- Dense Variational Reconstruction of Non-rigid Surfaces from Monocular Video2013
- Reconstructing 3D trajectories of independently moving objects using generic constraintsComputer Vision and Image Understanding, 2004
- A Closed-Form Solution to Non-rigid Shape and Motion Recovery2004