BronchoCopilot: Towards Autonomous Robotic Bronchoscopy via Multimodal Reinforcement Learning
Authors:
- 14 October 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 6923-6930
Abstract
Bronchoscopy plays a significant role in the early diagnosis and treatment of lung diseases. This process demands physicians to maneuver the flexible endoscope for reaching distal lesions, particularly requiring substantial expertise when examining the airways of the upper lung lobe. With the development of artificial intelligence and robotics, reinforcement learning (RL) method has been applied to the manipulation of interventional surgical robots. However, unlike human physicians who utilize multimodal information, most of the current RL methods rely on a single modality, limiting their performance. In this paper, we propose BronchoCopilot, a multimodal RL agent designed to acquire manipulation skills for autonomous bronchoscopy. BronchoCopilot specifically integrates images from the bronchoscope camera and estimated robot poses, aiming for a higher success rate within challenging airway environment. We employ auxiliary reconstruction tasks to compress multimodal data and utilize attention mechanisms to achieve an efficient latent representation of this data, serving as input for the RL module. This framework adopts a stepwise training and fine-tuning approach to mitigate the challenges of training difficulty. Our evaluation in the realistic simulation environment reveals that BronchoCopilot, by effectively harnessing multimodal information, attains a success rate of approximately 90% in fifth generation airways with consistent movements. Additionally, it demonstrates a robust capacity to adapt to diverse cases.Keywords
All Related Versions
Funding Information
- Chinese Academy of Sciences
This publication has 37 references indexed in Scilit:
- AI co-pilot bronchoscope robotNature Communications, 2024
- Learning-based autonomous vascular guidewire navigation without human demonstration in the venous system of a porcine liverInternational Journal of Computer Assisted Radiology and Surgery, 2022
- Deep Reinforcement Learning for Guidewire Navigation in Coronary Artery PhantomIEEE Access, 2021
- Reinforcement Learning with Attention that Works: A Self-Supervised Approach2019
- Flexible BronchoscopyClinics in Chest Medicine, 2018
- Deep Residual Learning for Image Recognition2016
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification2015
- SOFA: A Multi-Model Framework for Interactive Physical Simulation2012
- Bronchoscopy simulation: a brief reviewAdvances in Health Sciences Education, 2008
- Interventional robotic systems: Applications and technology state‐of‐the‐artMinimally Invasive Therapy & Allied Technologies, 2006