Multiview Integration Network for Multitask Robotic Surgical Scene Analysis
- 9 May 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 74 (00189456) , 1-13
Abstract
Surgical scene analysis holds a pivotal role in robot-assisted surgery. However, existing methods often suffer from single or little views, leading to erroneous scene analysis conclusions. To address these issues, a novel Multiview Integration Network (MVINet) is proposed, which comprehensively analyzes the surgical scene by integrating effective information from multiple views, including global, local, dynamic, and static views. As a multitask scene analysis network, MVINet could simultaneously perform semantic segmentation of surgical instruments and detection of instrument-tissue interaction. By designing a unique Global-Local Spatial Feature Memory module (G-LSFM), the joint information from globally and locally analyzed views following graph analysis simultaneously enhances the accuracy of multitask scene analysis. The Dynamic-Static Visual Feature Memory module (D-SVFM) introduces an innovative approach by simultaneously incorporating temporal feature from consecutive frames and static feature from single frame into the node features of the interaction reasoning network. The multiscale perspective of both features further enhances the ability of the module to analyze the complex scenes. Experimental results on a public and a private dataset demonstrate that our method achieves superior performance compared to other state-of-the-art methods for two crucial tasks in surgical scene analysis. In the instrument segmentation task, MVINet outperforms the second-best method by 1.73% and 0.55% in terms of mIoU scores. For the interaction detection task, MVINet surpasses the second-best method by 10.14% and 5.47% in mAP scores.Keywords
Funding Information
- National Key Research and Development Program of China (2022YFE0134700)
- National Natural Science Foundation of China (U22B2050)
- Science and Technology Program of Changsha (kq2102009)
This publication has 45 references indexed in Scilit:
- CGBA-Net: context-guided bidirectional attention network for surgical instrument segmentationInternational Journal of Computer Assisted Radiology and Surgery, 2023
- Instrument-tissue Interaction Quintuple Detection in Surgery Videos2022
- ISINet: An Instance-Based Approach for Surgical Instrument Segmentation2020
- Learning and Reasoning with the Graph Structure Representation in Robotic Surgery2020
- Graph neural networks: A review of methods and applicationsAI Open, 2020
- Learning Where to Look While Tracking Instruments in Robot-Assisted Surgery2019
- Learning Human-Object Interactions by Graph Parsing Neural Networks2018
- Concurrent Segmentation and Localization for Tracking of Surgical Instruments2017
- Simultaneous Recognition and Pose Estimation of Instruments in Minimally Invasive Surgery2017
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015