Developing the surgeon-machine interface: using a novel instance-segmentation framework for intraoperative landmark labelling
Authors:
Open Access
- 23 October 2023
- journal article
- research article
- Published by Frontiers Media SA in Frontiers in Surgery
- Vol. 10, 1259756
Abstract
Introduction: The utilisation of artificial intelligence (AI) augments intraoperative safety, surgical training, and patient outcomes. We introduce the term Surgeon-Machine Interface (SMI) to describe this innovative intersection between surgeons and machine inference. A custom deep computer vision (CV) architecture within a sparse labelling paradigm was developed, specifically tailored to conceptualise the SMI. This platform demonstrates the ability to perform instance segmentation on anatomical landmarks and tools from a single open spinal dural arteriovenous fistula (dAVF) surgery video dataset.Methods: Our custom deep convolutional neural network was based on SOLOv2 architecture for precise, instance-level segmentation of surgical video data. Test video consisted of 8520 frames, with sparse labelling of only 133 frames annotated for training. Accuracy and inference time, assessed using F1-score and mean Average Precision (mAP), were compared against current state-of-the-art architectures on a separate test set of 85 additionally annotated frames.Results: Our SMI demonstrated superior accuracy and computing speed compared to these frameworks. The F1-score and mAP achieved by our platform were 17% and 15.2% respectively, surpassing MaskRCNN (15.2%, 13.9%), YOLOv3 (5.4%, 11.9%), and SOLOv2 (3.1%, 10.4%). Considering detections that exceeded the Intersection over Union threshold of 50%, our platform achieved an impressive F1-score of 44.2% and mAP of 46.3%, outperforming MaskRCNN (41.3%, 43.5%), YOLOv3 (15%, 34.1%), and SOLOv2 (9%, 32.3%). Our platform demonstrated the fastest inference time (88ms), compared to MaskRCNN (90ms), SOLOV2 (100ms), and YOLOv3 (106ms). Finally, the minimal amount of training set demonstrated a good generalisation performance –our architecture successfully identified objects in a frame that were not included in the training or validation frames, indicating its ability to handle out-of-domain scenarios.Discussion: We present our development of an innovative intraoperative SMI to demonstrate the future promise of advanced CV in the surgical domain. Through successful implementation in a microscopic dAVF surgery, our framework demonstrates superior performance over current state-of-the-art segmentation architectures in intraoperative landmark guidance with high sample efficiency, representing the most advanced AI-enabled surgical inference platform to date. Our future goals include transfer learning paradigms for scaling to additional surgery types, addressing clinical and technical limitations for performing real-time decoding, and ultimate enablement of a real-time neurosurgical guidance platform.Keywords
This publication has 25 references indexed in Scilit:
- Label-free brain tumor imaging using Raman-based methodsJournal of Neuro-Oncology, 2021
- Artificial Intelligence for Surgical SafetyAnnals of Surgery, 2020
- Machine Learning for Surgical Phase RecognitionAnnals of Surgery, 2020
- Artificial Intelligence for Intraoperative GuidanceAnnals of Surgery, 2020
- Development of an artificial intelligence system using deep learning to indicate anatomical landmarks during laparoscopic cholecystectomySurgical Endoscopy, 2020
- Detection of colorectal adenomas with a real-time computer-aided system (ENDOANGEL): a randomised controlled studyThe Lancet Gastroenterology & Hepatology, 2020
- Explainable Artificial Intelligence for Safe Intraoperative Decision SupportJAMA Surgery, 2019
- Computer Vision Analysis of Intraoperative VideoAnnals of Surgery, 2019
- Artificial Intelligence in Surgery: Promises and PerilsAnnals of Surgery, 2018
- Detecting Surgical Tools by Modelling Local Appearance and Global ShapeIEEE Transactions on Medical Imaging, 2015