Real-Time and High-Accuracy Switchable Stereo Depth Estimation Method Utilizing Self-Supervised Online Learning Mechanism for MIS
- 5 August 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 73 (00189456) , 1-13
Abstract
In minimally invasive surgery (MIS), clinicians often rely on 2-D laparoscopic images to assess the size and distances between internal structures. However, this subjective estimation introduces significant uncertainty and can increase surgical risks while reducing efficiency. Modern 3-D laparoscopes offer improved stereoscopic perception and can incorporate stereo depth estimation methods for quantitative analysis. However, existing methods struggle with real-time and high-accuracy demands in diverse surgical scenarios. To address this issue, we propose a novel intraoperative stereo depth estimation framework termed metainitialized online learning (MIOL), aiming to assist surgeons in quantitatively controlling surgical targets during the procedure. This framework features two switchable modes and does not require annotated data. One mode enables rapid depth recovery through surgical videos, providing real-time 3-D reconstruction to help surgeons understand in vivo structures. The other mode achieves high-precision measurements of critical tissues in fixed frames, assisting in surgical decision-making. Our approach employs self-supervised adaptation to train a model specific to each stereo image, eliminating the need for generalization and achieving outstanding accuracy. The framework establishes a lightweight network that converges rapidly under self-supervised losses and incorporates meta-learning pretraining, sparse optical flow guidance, and effective region identification to ensure speed and accuracy. Extensive experiments on two public datasets demonstrate the superiority of our method over existing approaches. Furthermore, we implement an intraoperative measurement system and conduct clinical trials, confirming its practical utility. The code is available at https://github.com/Darcy-vision/MIOL .Keywords
Funding Information
- National Natural Science Foundation of China (62133004, 72293585, 72188101)
- Anhui Provincial Natural Science Foundation (2108085J33)
- Anhui Provincial Major Science and Technology Project (202203a05020010)
- Fundamental Research Funds for the Central Universities (JZ2022HGPA0311, JZ2023HGQA0125)
This publication has 32 references indexed in Scilit:
- FRSR: Framework for real-time scene reconstruction in robot-assisted minimally invasive surgeryComputers in Biology and Medicine, 2023
- Artificial intelligence and automation in endoscopy and surgeryNature Reviews Gastroenterology & Hepatology, 2022
- Self-supervised Depth Estimation in Laparoscopic Image Using 3D Geometric Consistency2022
- StaSiS-Net: A stacked and siamese disparity estimation network for depth reconstruction in modern 3D laparoscopyMedical Image Analysis, 2022
- Unsupervised Scale-Consistent Depth Learning from VideoInternational Journal of Computer Vision, 2021
- An edge-aware based adaptive multi-feature set extraction for stereo matching of binocular imagesJournal of Ambient Intelligence and Humanized Computing, 2021
- StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction2018
- Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Multi-view Scene Flow Estimation: A View Centered Variational ApproachInternational Journal of Computer Vision, 2012