Artificial intelligence-supported segmentation of cardiac anatomy in open-heart surgery videos
Authors:
Open Access
- 6 August 2026
- journal article
- research article
- Published by Springer Nature in Journal of Robotic Surgery
- Vol. 20 (1) , 801
Abstract
Accurate identification of anatomical structures is essential for safe congenital cardiac surgery and for the development of assistive robotic systems. However, scalable annotation of open-heart surgical video remains a major challenge due to dynamic tissue motion, occlusion, and anatomical variability. The aim of this study was to develop and evaluate a human-in-the-loop segmentation and tracking pipeline for congenital open-heart surgery videos. A dataset of 72 annotated video clips comprising 27,461 frames was created from routine recordings of congenital cardiac surgery. Independent tracking evaluation was performed on 6 video clips (2,400 frames) from three surgical cases that were not used for training or validation. A hybrid framework combining fine-tuned Segment Anything Model 2 (SAM 2) for propagation-based tracking and YOLOv11 for detection-based monitoring was implemented and evaluated. Fine-tuning improved temporal tracking performance compared with the pretrained model, increasing the mean IoU from 82.1% to 92.2% and the proportion of frames with IoU ≥ 90% from 48.2% to 72.3% on the independent evaluation dataset. A graphical user interface enabled efficient dataset creation, reducing annotation time by approximately 40-fold compared with manual tracing. This study demonstrates that domain-adapted foundation-model tracking can provide robust anatomical tracking in a dynamic open surgical environment and offers a scalable framework for future assistive and robotic cardiac applications.Keywords
Funding Information
- Lunds Universitet
This publication has 24 references indexed in Scilit:
- A fine-tuned foundational model SurgiSAM2 for surgical video anatomy segmentation and detectionScientific Reports, 2025
- Deep Learning Analysis of Surgical Video Recordings to Assess Nontechnical SkillsJAMA Network Open, 2024
- A systematic review of annotation for surgical process model analysis in minimally invasive surgery based on videoSurgical Endoscopy, 2023
- Video-based formative and summative assessment of surgical tasks using deep learningScientific Reports, 2023
- Evaluation of Deep Learning Models for Identifying Surgical Actions and Measuring PerformanceJAMA Network Open, 2020
- Analysis of Human Performance Deficiencies Associated With Surgical Adverse EventsJAMA Network Open, 2019
- Coordinated control of a dual-arm robot for surgical instrument sorting tasksRobotics and Autonomous Systems, 2019
- A software-based tool for video motion tracking in the surgical skills assessment landscapeSurgical Endoscopy, 2018
- A survey on deep learning in medical image analysisMedical Image Analysis, 2017
- Digital image analysis outperforms manual biomarker assessment in breast cancerLaboratory Investigation, 2016