Grounding Surgical Action Triplets with Instrument Instance Segmentation: A Dataset and Target-Aware Fusion Approach
Preprint
Abstract
Understanding surgical instrument-tissue interactions requires not only identifying which instrument performs which action on which anatomical target, but also grounding these interactions spatially within the surgical scene. Existing surgical action triplet recognition methods are limited to learning from frame-level classification, failing to reliably link actions to specific instrument instances.Previous attempts at spatial grounding have primarily relied on class activation maps, which lack the precision and robustness required for detailed instrument-tissue interaction analysis.To address this gap, we propose grounding surgical action triplets with instrument instance segmentation, or triplet segmentation for short, a new unified task which produces spatially grounded outputs.We start by presenting CholecTriplet-Seg, a large-scale dataset containing over 30,000 annotated frames, linking instrument instance masks with action verb and anatomical target annotations, and establishing the first benchmark for strongly supervised, instance-level triplet grounding and evaluation.To learn triplet segmentation, we propose TargetFusionNet, a novel architecture that extends Mask2Former with a target-aware fusion mechanism to address the challenge of accurate anatomical target prediction by fusing weak anatomy priors with instrument instance queries.Evaluated across recognition, detection, and triplet segmentation metrics, TargetFusionNet consistently improves performance over existing baselines, demonstrating that strong instance supervision combined with weak target priors significantly enhances the accuracy and robustness of surgical action understanding.Triplet segmentation establishes a unified framework for spatially grounding surgical action triplets. The proposed benchmark and architecture pave the way for more interpretable, surgical scene understanding.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- ROBUST-MIPS: A Combined Skeletal Pose and Instance Segmentation Dataset for Laparoscopic Surgical InstrumentsScientific Data, 2026
- Comparative validation of surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation in endoscopy: Results of the PhaKIR 2024 challengeMedical Image Analysis, 2026
- Pixel-wise recognition for holistic surgical scene understandingMedical Image Analysis, 2025
- CholecInstanceSeg: A Tool Instance Segmentation Dataset for Laparoscopic SurgeryScientific Data, 2025
- Deep learning for surgical instrument recognition and segmentation in robotic-assisted surgeries: a systematic reviewArtificial Intelligence Review, 2024
- EndoViT: pretraining vision transformers on a large collection of endoscopic imagesInternational Journal of Computer Assisted Radiology and Surgery, 2024
- CholecTriplet2022: Show me a tool and tell me the triplet — An endoscopic vision challenge for surgical action triplet detectionMedical Image Analysis, 2023
- Rendezvous in time: an attention-based temporal fusion approach for surgical triplet recognitionInternational Journal of Computer Assisted Radiology and Surgery, 2023
- Automated identification of critical structures in laparoscopic cholecystectomyInternational Journal of Computer Assisted Radiology and Surgery, 2022
- Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videosMedical Image Analysis, 2022