AUGMENTED REALITY-BASED NASAL ENDOSCOPE VIDEO RECONSTRUCTION AND REGISTRATION
- 26 October 2023
- journal article
- research article
- Published by World Scientific Pub Co Pte Ltd in Journal of Mechanics in Medicine and Biology
- Vol. 23 (09)
Abstract
Neuroendoscopic surgery is a minimally invasive surgical technique commonly used to remove a tumor through the patient’s mouth or nose, which requires the surgeon to avoid important neural and vascular structures. Augmented reality-based surgical navigation can provide surgeons with more information and visual aids, clarifying the structure and details of the surgical area. It requires real-time registration of intraoperative endoscopic video with pre-operative CT models. However, 3D reconstruction and camera tracking from nasal endoscopic video are challenging due to the narrow nasal cavity and the lack of texture. Besides, the nasal endoscopic datasets that can be used for deep learning-based depth estimation are scarce and hard to elaborate. To this end, a style-augmented module (SAM) is proposed in this study to minimize discrepancies between different endoscopic datasets. An unsupervised approach was trained to generate depth maps and camera motion paths, which were used to reconstruct the nasal scene’s 3D model. The obtained reconstruction was aligned with a pre-operative CT model for augmented reality-based surgical navigation. The results showed that the proposed SAM improved the generalization of the depth estimation model on nasal endoscopy data. The proposed approach to combining augmented reality technology with surgical navigation is considered instrumental in furnishing surgeons with richer information on the nasal endoscopic surgical scene.Keywords
This publication has 20 references indexed in Scilit:
- Generalizable Cross-modality Medical Image Segmentation via Style Augmentation and Dual Normalization2022
- The Challenge of Augmented Reality in Surgery2020
- Endoscopic Navigation in the Absence of CT Imaging2018
- Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction2018
- Unsupervised Monocular Depth Estimation with Left-Right Consistency2017
- The status of augmented reality in laparoscopic surgery as of 2016Medical Image Analysis, 2017
- Structure-from-Motion Revisited2016
- Multicore bundle adjustment2011
- Real-Time Stereo Reconstruction in Robotically Assisted Minimally Invasive Surgery2010
- Soft-Tissue Motion Tracking and Structure Estimation for Robotic Assisted MIS Procedures2005