Anton Deguet

Virtual simulations have served as important tools for designing and testing robotic frameworks in various domains. Increasingly, they have been used to train and experiment in surgical robotics applications. Having seen the success of an open simulation environment for robot-assisted surgical suturing in the Asynchronous Multi-Body Framework (AMBF), we sought to reproduce the surgical suturing environment from AMBF in NVIDIA Isaac Sim, a simulator with impressive visual fidelity and GPU-accelerated computation capabilities. We present a virtual scene for robotic suturing comprising two robotic manipulators, an endoscope, two surgical phantoms and a needle attached to a thread. The thread is modeled using three different techniques. This work also provides interactive manipulation of the virtual scene via haptic input devices using the Robot Operating System interface. Although Isaac Sim possesses impressive and realistic visuals, AMBF outperforms in speed by up to 15 frames per second, depending on the task. AMBF also responded faster than Isaac Sim by approximately 0.1 s when using a haptic input device. In this context, our objective is to provide a comprehensive overview of the capabilities and limitations of NVIDIA Isaac Sim in the surgical robotics challenge relative to AMBF.

The growing global shortage of skilled surgeons underscores the need for intelligent, assistive technologies in the operating room. To address this challenge, we introduce ImitateCholec, a publicly available dataset specifically designed to advance autonomous robotic systems during the critical clipping and cutting phase of laparoscopic cholecystectomy. The dataset comprises over 18,000 demonstrations from 34 ex vivo porcine cholecystectomies, totaling approximately 20 hours of data. Each clipping and cutting phase recorded in the dataset is segmented into 17 distinct surgical tasks. ImitateCholec uniquely integrates endoscopic videos captured from multiple camera perspectives with comprehensive kinematic data acquired through the da Vinci Research Kit. Both optimal demonstration executions and recovery maneuvers were systematically recorded, enabling the training of imitation learning models capable of robustly addressing real-world surgical variability. Primarily, ImitateCholec facilitates imitation learning for long-horizon surgical workflow execution, significantly advancing the development of autonomous robotic systems toward achieving phase-level autonomy and, ultimately, full procedural autonomy. Additional supported applications include surgical workflow modeling, error recognition, and surgical tool pose estimation.

Cancer resection surgery is unsuccessful if tumor tissue is left behind in the surgical cavity. Identifying the residual cancer requires additional imaging or postoperative histological analysis. Photoacoustic imaging can be used to image both the surface and depths of the resection cavity; however, its performance hinges on consistent probe placement and stable acoustic and optical coupling. As intra-cavity deployment of photoacoustic imaging is largely uncharted, several potential embodiments warrant rigorous investigation. We address this need with an open-source robotic testbed for intraoperative tumor-bed inspection using photoacoustic imaging. The platform integrates the da Vinci Research Kit, depth imaging, and electromagnetic tracking to automate cavity scanning and maintain repeatable probe trajectories. Using tissue-mimicking phantoms, we (i) demonstrate a novel imaging embodiment for photoacoustic tumor-bed inspection and (ii) show how this testbed can be used to investigate and optimize tumor bed inspection strategies and configurations. This study establishes the feasibility of detecting and mapping residual cancer within a simulated surgical cavity. The primary contribution is the testbed itself, designed for integration with existing surgical navigation workflows and rapid prototyping. This testbed serves as an essential foundation for systematic evaluation of photoacoustic, robot-assisted strategies for improving intraoperative margin assessment.

Surgeon shortages have driven significant recent advances in robotic surgical assistance. To progress toward surgical automation, we address transoral central airway obstruction (CAO) removal, selecting this procedure for its relatively constrained environment and clear clinical objectives. Prior work has demonstrated successful CAO removal with single-port, minimally invasive robotic systems, making it a promising target for autonomy. We present a new autonomous system designed for CAO removal that integrates a learning-based tissue retraction policy with a model-based resection planner, deployed on a bimanual robotic platform with vision guidance. While our long-term motivation is minimally invasive intervention, we evaluate our approach in an open-surgery setting, enabling rapid development and testing. We evaluate our system in 12 ex vivo CAO removal procedures running at three distinct levels of autonomy. Our results indicate that the learned retraction policy performs similarly or better than a manually designed heuristic, with a mean surgeon-rated score (1-10) of 8.0± 1.6 compared to 6.8± 2.5, and a mean percent tumor volume removal of 113.9± 18.8% compared to 111.9± 13.3%. The model-based resections performed best when supervised, resulting in less tissue damage and better cutting accuracy. Furthermore, our approach produced clinically successful outcomes in all eight supervised autonomous trials. Notably, in one case, the system achieved complete tumor removal without any human intervention. Our overall findings underscore the importance of high-level supervision, revealing targets for future automation.

Robot-assisted minimally invasive surgery, such as through the da Vinci systems, improves precision and patient outcomes. However, da Vinci systems prior to da Vinci 5 lacked direct force-sensing capabilities, forcing surgeons to operate without the haptic feedback they get through laparoscopy. Our prior work restored force sensing through machine learning-based force estimation for an open-source surgical robotics research platform, the da Vinci Research Kit (dVRK) Classic. This study extends our previous method to the newer dVRK system, the dVRK-Si. Additionally, we benchmark the performance of the learning-based algorithm against baseline methods (which make simplifying assumptions on the torque) to study how the two systems differ. In both systems, the learning-based method outperforms baselines, but the difference is much larger in the dVRK-Si. Nonetheless, dVRK-Si force estimation accuracy lags behind the dVRK Classic, with Root Mean Square Error (RMSE) 2 to 3 times higher. Further analysis reveals suboptimal PID control in the dVRK-Si. We hypothesize that this is because, unlike the dVRK Classic, the dVRK-Si is not mechanically balanced and exhibits more complex internal dynamics. This study advances the understanding of learning-based force estimation and is the first work to implement learning-based dynamics estimation of the new dVRK-Si system.

Existing tracheal tumor resection methods often lack the precision required for effective airway clearance, and robotic advancements offer new potential for autonomous resection. We present a vision-guided, autonomous approach for palliative resection of tracheal tumors. This system models the tracheal surface with a fifth-degree polynomial to plan tool trajectories, while a custom Faster R-CNN segmentation pipeline identifies the trachea and tumor boundaries. The electrocautery tool angle is optimized using handheld surgical demonstrations, and trajectories are planned to maintain a 1[ ]mm safety clearance from the tracheal surface. We validated the workflow successfully in five consecutive experiments on ex-vivo animal tissue models, successfully clearing the airway obstruction without trachea perforation in all cases (with more than 90% volumetric tumor removal). These results support the feasibility of an autonomous resection platform, paving the way for future developments in minimally-invasive autonomous resection.

Research on autonomous surgery has largely focused on simple task automation in controlled environments. However, real-world surgical applications demand dexterous manipulation over extended durations and robust generalization to the inherent variability of human tissue. These challenges remain difficult to address using existing logic-based or conventional end-to-end learning strategies. To address this gap, we propose a hierarchical framework for performing dexterous, long-horizon surgical steps. Our approach uses a high-level policy for task planning and a low-level policy for generating low-level trajectories. The high-level planner plans in language space, generating task-level or corrective instructions that guide the robot through the long-horizon steps and help recover from errors made by the low-level policy. We validated our framework through ex vivo experiments on cholecystectomy, a commonly practiced minimally invasive procedure, and conducted ablation studies to evaluate key components of the system. Our method achieves a 100% success rate across eight different ex vivo gallbladders, operating fully autonomously without human intervention. The hierarchical approach improved the policy’s ability to recover from suboptimal states that are inevitable in the highly dynamic environment of realistic surgical applications. This work demonstrates step-level autonomy in a surgical procedure, marking a milestone toward clinical deployment of autonomous surgical systems.

Autonomous robotic surgery has the potential to achieve better patient outcomes and greater efficiency than manual approaches by leveraging precise robotic motion for task execution. Unlike articulated robots, tendon-driven surgical robots suffer from lower motion precision due to the hysteresis and friction inherent in their cable-pulley design. These nonlinear mechanical effects complicate precise end-effector control, posing challenges to autonomous task performance and safety concerns. In this paper, we present a calibration workflow using machine learning models to enhance the positioning accuracy of a tendon-driven surgical robot, evaluated on the real da Vinci Research Kit Si (dVRK-Si). An accuracy analysis was carried out for the dVRK-Si and a transformer-based model was trained to compensate for the nonlinear effects of the system, enhancing system positioning accuracy. The calibrated system was evaluated on trajectory-following tasks in Cartesian space, demonstrating 51.37% and 45.87% reduction in position and orientation errors. Additionally, we tested the models’ transferability with two uncalibrated surgical instruments and reported accuracy improvements over the first-generation dVRK-Classic.

Image-guided robotic interventions represent a transformative frontier in surgery, blending advanced imaging and robotics for improved precision and outcomes. This paper addresses the critical need for integrating open-source platforms to enhance situational awareness in image-guided robotic research. We present an open-source toolkit, named ENTRI, that seamlessly combines a physics-based constraint formulation framework, AMBF, with a state-of-the-art imaging platform application, 3D Slicer. ENTRI facilitates the creation of highly customizable interactive digital twins, that incorporate processing and visualization of medical imaging, robot kinematics, and scene dynamics for real-time robot control. Through a feasibility study, we showcase real-time synchronization of a physical robotic interventional environment in both 3D Slicer and AMBF, highlighting low-latency updates and improved visualization. The source code and supplementary materials for this study are available here.