“Deep-Onto” network for surgical workflow and context recognition
- 16 November 2018
- journal article
- research article
- Published by Springer Nature in International Journal of Computer Assisted Radiology and Surgery
- Vol. 14 (4) , 685-696
Abstract
Purpose Surgical workflow recognition and context-aware systems could allow better decision making and surgical planning by providing the focused information, which may eventually enhance surgical outcomes. While current developments in computer-assisted surgical systems are mostly focused on recognizing surgical phases, they lack recognition of surgical workflow sequence and other contextual element, e.g., “Instruments.” Our study proposes a hybrid approach, i.e., using deep learning and knowledge representation, to facilitate recognition of the surgical workflow. Methods We implemented “Deep-Onto” network, which is an ensemble of deep learning models and knowledge management tools, ontology and production rules. As a prototypical scenario, we chose robot-assisted partial nephrectomy (RAPN). We annotated RAPN videos with surgical entities, e.g., “Step” and so forth. We performed different experiments, including the inter-subject variability, to recognize surgical steps. The corresponding subsequent steps along with other surgical contexts, i.e., “Actions,” “Phase” and “Instruments,” were also recognized. Results The system was able to recognize 10 RAPN steps with the prevalence-weighted macro-average (PWMA) recall of 0.83, PWMA precision of 0.74, PWMA F1 score of 0.76, and the accuracy of 74.29% on 9 videos of RAPN. Conclusion We found that the combined use of deep learning and knowledge representation techniques is a promising approach for the multi-level recognition of RAPN surgical workflow.Keywords
Funding Information
- Horizon 2020 Framework Programme (H2020-ICT-2016-732515)
This publication has 24 references indexed in Scilit:
- Development of an intelligent surgical training system for ThoracentesisArtificial Intelligence in Medicine, 2018
- EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic VideosIEEE Transactions on Medical Imaging, 2016
- LapOntoSPM: an ontology for laparoscopic surgeries and its application to surgical phase recognitionInternational Journal of Computer Assisted Radiology and Surgery, 2015
- Cognitive skills assessment during robot‐assisted surgery: separating the wheat from the chaffBJU International, 2014
- Technique and Outcomes of Robot-assisted Retroperitoneoscopic Partial Nephrectomy: A Multicenter StudyEuropean Urology, 2014
- Cancer incidence and mortality patterns in Europe: Estimates for 40 countries in 2012European Journal of Cancer, 2013
- Identifying the risk of disease progression after surgery for localized renal cell carcinomaBJU International, 2010
- Modeling and Segmentation of Surgical Workflow from Laparoscopic Video2010
- How do surgeons make intraoperative decisions?Quality and Safety in Health Care, 2007
- Automatic Detection and Segmentation of Robot-Assisted Surgical Motions2005