Zero-shot Monocular Metric Depth for Endoscopic Images
Preprint
Abstract
Monocular relative and metric depth estimation has seen a tremendous boost in the last few years due to the sharp advancements in foundation models and in particular transformer based networks. As we start to see applications to the domain of endoscopic images, there is still a lack of robust benchmarks and high-quality datasets in that area. This paper addresses these limitations by presenting a comprehensive benchmark of state-of-the-art (metric and relative) depth estimation models evaluated on real, unseen endoscopic images, providing critical insights into their generalisation and performance in clinical scenarios. Additionally, we introduce and publish a novel synthetic dataset (EndoSynth) of endoscopic surgical instruments paired with ground truth metric depth and segmentation masks, designed to bridge the gap between synthetic and real-world data. We demonstrate that fine-tuning depth foundation models using our synthetic dataset boosts accuracy on most unseen real data by a significant margin. By providing both a benchmark and a synthetic dataset, this work advances the field of depth estimation for endoscopic images and serves as an important resource for future research. Project page, EndoSynth dataset and trained weights are available at https://github.com/TouchSurgery/EndoSynth.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- A narrative review of the Medtronic Hugo RAS and technical comparison with the Intuitive da Vinci robotic surgical systemJournal of Robotic Surgery, 2024
- Depth Anything V22024
- Unifying Flow, Stereo and Depth EstimationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2023
- Learning how to robustly estimate camera pose in endoscopic videosInternational Journal of Computer Assisted Radiology and Surgery, 2023
- A vision transformer for decoding surgeon activity from surgical videosNature Biomedical Engineering, 2023
- Self-Supervised monocular depth and ego-Motion estimation in endoscopy: Appearance flow to the rescueMedical Image Analysis, 2022
- Endo-Depth-and-Motion: Reconstruction and Tracking in Endoscopic Videos Using Depth Networks and Photometric ConstraintsIEEE Robotics and Automation Letters, 2021
- Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset TransferIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
- Albumentations: Fast and Flexible Image AugmentationsInformation, 2020
- Self-Supervised Siamese Learning on Stereo Image Pairs for Depth Estimation in Robotic Surgery2017