Large language model-augmented implicit surgical video review

Authors:
Preprint
Publisher website
Google Scholar
Abstract
Surgical video interpretation is a promising medical artificial intelligence application. However, no existing video annotation method preserves the spatiotemporal complexity of surgeon reasoning. Here we show that verbal reasoning and visual attention can be converted into structured, machine-actionable records of intraoperative behaviours. Our method decomposes transcribed verbal commentary into video-anchored semantic feedback chunks, which are classified via a large language model, with spatial grounding to surgical scenes via eyegaze or cursor tracking. We demonstrate method validity and scalability on structured and unstructured annotation tasks. For quality feedback on full-length colorectal procedures, the method reached near-human fidelity for chunking (cosine similarity: 0.95±0.01) and semantic classification across observations (Cohen’s κ: 0.71±0.07) and evaluative triggers (Cohen’s κ: 0.67±0.14), with excellent usability ratings. For structured critical view of safety assessment in laparoscopic cholecystectomy, implicit annotation yielded excellent agreement with explicit reviewer ratings (Cohen’s κ: 0.83, 0.49 and 0.81 across three criteria). We anticipate this method will advance surgical data science by enabling scalable construction of meaningfully annotated surgical video datasets.
Funding Information
  • National Institute on Aging (K23 AG071945)
  • Bundesministerium für Forschung, Technologie und Raumfahrt
  • Deutsches Krebsforschungszentrum (CoBot 2.0)
  • Jung-Stiftung für Wissenschaft und Forschung (Jung Career Advancement Award)
  • Central Indiana Corporate Partnership AnalytiXIN Initiative (NA)
  • Indiana Clinical and Translational Sciences Institute (UM1TR004402 (NIH NCATS))

This publication has 32 references indexed in Scilit: