A Surface-normal Based Neural Framework for Colonoscopy Reconstruction
Preprint
Abstract
Reconstructing a 3D surface from colonoscopy video is challenging due to illumination and reflectivity variation in the video frame that can cause defective shape predictions. Aiming to overcome this challenge, we utilize the characteristics of surface normal vectors and develop a two-step neural framework that significantly improves the colonoscopy reconstruction quality. The normal-based depth initialization network trained with self-supervised normal consistency loss provides depth map initialization to the normal-depth refinement module, which utilizes the relationship between illumination and surface normals to refine the frame-wise normal and depth predictions recursively. Our framework's depth accuracy performance on phantom colonoscopy data demonstrates the value of exploiting the surface normals in colonoscopy reconstruction, especially on en face views. Due to its low depth error, the prediction result from our framework will require limited post-processing to be clinically applicable for real-time colonoscopy reconstruction.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- Colonoscopy 3D video dataset with paired depth from 2D-3D registrationMedical Image Analysis, 2023
- RNNSLAM: Reconstructing the 3D colon to visualize missing regions during a colonoscopyMedical Image Analysis, 2021
- Dense Depth Estimation in Monocular Endoscopy With Self-Supervised Learning MethodsIEEE Transactions on Medical Imaging, 2019
- Mesh-Based Computation for Solving Photometric Stereo With Near Point LightingIEEE Computer Graphics and Applications, 2019
- Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial TrainingIEEE Transactions on Medical Imaging, 2018
- Direct Sparse OdometryIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017