Video-based assessment of intraoperative surgical skill
- 30 May 2022
- journal article
- research article
- Published by Springer Nature in International Journal of Computer Assisted Radiology and Surgery
- Vol. 17 (10) , 1801-1811
Abstract
Purpose Surgeons’ skill in the operating room is a major determinant of patient outcomes. Assessment of surgeons’ skill is necessary to improve patient outcomes and quality of care through surgical training and coaching. Methods for video-based assessment of surgical skill can provide objective and efficient tools for surgeons. Our work introduces a new method based on attention mechanisms and provides a comprehensive comparative analysis of state-of-the-art methods for video-based assessment of surgical skill in the operating room. Methods Using a dataset of 99 videos of capsulorhexis, a critical step in cataract surgery, we evaluated image feature-based methods and two deep learning methods to assess skill using RGB videos. In the first method, we predict instrument tips as keypoints and predict surgical skill using temporal convolutional neural networks. In the second method, we propose a frame-wise encoder (2D convolutional neural network) followed by a temporal model (recurrent neural network), both of which are augmented by visual attention mechanisms. We computed the area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and predictive values through fivefold cross-validation. Results To classify a binary skill label (expert vs. novice), the range of AUC estimates was 0.49 (95% confidence interval; CI = 0.37 to 0.60) to 0.76 (95% CI = 0.66 to 0.85) for image feature-based methods. The sensitivity and specificity were consistently high for none of the methods. For the deep learning methods, the AUC was 0.79 (95% CI = 0.70 to 0.88) using keypoints alone, 0.78 (95% CI = 0.69 to 0.88) and 0.75 (95% CI = 0.65 to 0.85) with and without attention mechanisms, respectively. Conclusion Deep learning methods are necessary for video-based assessment of surgical skill in the operating room. Attention mechanisms improved discrimination ability of the network. Our findings should be evaluated for external validity in other datasets.Keywords
All Related Versions
Funding Information
- National Eye Institute (R01EY033065)
This publication has 29 references indexed in Scilit:
- Crowdsourcing Assessment of Surgeon Dissection of Renal Artery and Vein During Robotic Partial Nephrectomy: A Novel Approach for Quantitative Assessment of Surgical PerformanceJournal of Endourology, 2016
- Automated Assessment of Surgical Skills Using Frequency Analysis2015
- Crowdsourcing to Assess Surgical SkillJAMA Surgery, 2015
- Crowd-sourced assessment of surgical skills in cricothyrotomy procedureJournal of Surgical Research, 2015
- Surgical Skill and Complication Rates after Bariatric SurgeryNew England Journal of Medicine, 2013
- Self-Assessment of Technical Skill in Surgery: The Need for Expert FeedbackThe Annals of The Royal College of Surgeons of England, 2008
- Human Detection Using Oriented Histograms of Flow and Appearance2006
- On Space-Time Interest PointsInternational Journal of Computer Vision, 2005
- Understanding inverse document frequency: on theoretical arguments for IDFJournal of Documentation, 2004
- Long Short-Term MemoryNeural Computation, 1997