Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting
Preprint
Abstract
In the realm of robot-assisted minimally invasive surgery, dynamic scene reconstruction can significantly enhance downstream tasks and improve surgical outcomes. Neural Radiance Fields (NeRF)-based methods have recently risen to prominence for their exceptional ability to reconstruct scenes. Nonetheless, these methods are hampered by slow inference, prolonged training, and substantial computational demands. Additionally, some rely on stereo depth estimation, which is often infeasible due to the high costs and logistical challenges associated with stereo cameras. Moreover, the monocular reconstruction quality for deformable scenes is currently inadequate. To overcome these obstacles, we present Endo-4DGS, an innovative, real-time endoscopic dynamic reconstruction approach that utilizes 4D Gaussian Splatting (GS) and requires no ground truth depth data. This method extends 3D GS by incorporating a temporal component and leverages a lightweight MLP to capture temporal Gaussian deformations. This effectively facilitates the reconstruction of dynamic surgical scenes with variable conditions. We also integrate Depth-Anything to generate pseudo-depth maps from monocular views, enhancing the depth-guided reconstruction process. Our approach has been validated on two surgical datasets, where it can effectively render in real-time, compute efficiently, and reconstruct with remarkable accuracy. These results underline the vast potential of Endo-4DGS to improve surgical assistance.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering2024
- Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data2024
- Surgical-DINO: adapter learning of foundation models for depth estimation in endoscopic surgeryInternational Journal of Computer Assisted Radiology and Surgery, 2024
- Neural LerPlane Representations for Fast 4D Reconstruction of Deformable Tissues2023
- 3D Gaussian Splatting for Real-Time Radiance Field RenderingACM Transactions on Graphics, 2023
- Learning how to robustly estimate camera pose in endoscopic videosInternational Journal of Computer Assisted Radiology and Surgery, 2023
- Self-Supervised monocular depth and ego-Motion estimation in endoscopy: Appearance flow to the rescueMedical Image Analysis, 2022
- EndoSLAM dataset and an unsupervised monocular visual odometry and depth estimation approach for endoscopic videosMedical Image Analysis, 2021
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis2020
- Differentiable surface splatting for point-based geometry processingACM Transactions on Graphics, 2019