Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting
Preprint
Abstract
In the realm of robot-assisted minimally invasive surgery, dynamic scene reconstruction can significantly enhance downstream tasks and improve surgical outcomes. Neural Radiance Fields (NeRF)-based methods have recently risen to prominence for their exceptional ability to reconstruct scenes but are hampered by slow inference speed, prolonged training, and inconsistent depth estimation. Some previous work utilizes ground truth depth for optimization but is hard to acquire in the surgical domain. To overcome these obstacles, we present Endo-4DGS, a real-time endoscopic dynamic reconstruction approach that utilizes 3D Gaussian Splatting (GS) for 3D representation. Specifically, we propose lightweight MLPs to capture temporal dynamics with Gaussian deformation fields. To obtain a satisfactory Gaussian Initialization, we exploit a powerful depth estimation foundation model, Depth-Anything, to generate pseudo-depth maps as a geometry prior. We additionally propose confidence-guided learning to tackle the ill-pose problems in monocular depth estimation and enhance the depth-guided reconstruction with surface normal constraints and depth regularization. Our approach has been validated on two surgical datasets, where it can effectively render in real-time, compute efficiently, and reconstruct with remarkable accuracy.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Depth-Regularized Optimization for 3D Gaussian Splatting in Few-Shot Images2024
- 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering2024
- Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data2024
- Surgical-DINO: adapter learning of foundation models for depth estimation in endoscopic surgeryInternational Journal of Computer Assisted Radiology and Surgery, 2024
- Neural LerPlane Representations for Fast 4D Reconstruction of Deformable Tissues2023
- Learning how to robustly estimate camera pose in endoscopic videosInternational Journal of Computer Assisted Radiology and Surgery, 2023
- Self-Supervised monocular depth and ego-Motion estimation in endoscopy: Appearance flow to the rescueMedical Image Analysis, 2022
- EndoSLAM dataset and an unsupervised monocular visual odometry and depth estimation approach for endoscopic videosMedical Image Analysis, 2021
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis2020
- Differentiable surface splatting for point-based geometry processingACM Transactions on Graphics, 2019