Zerui Wang

Surgical robots capable of autonomously performing various tasks could enhance efficiency and augment human productivity in addressing clinical needs. Although current solutions have automated specific actions within defined contexts, they are challenging to generalize across diverse environments in general surgery. Embodied intelligence enables general-purpose robot learning with applications for daily tasks, yet its application in the medical domain remains limited. We introduced an open-source surgical embodied intelligence simulator for an interactive environment to develop reinforcement learning methods for minimally invasive surgical robots. Using such embodied artificial intelligence, this study further addresses surgical task automation, enabling zero-shot transfer of simulation-trained policies to real-world scenarios. The proposed method encompasses visual parsing, a perceptual regressor, policy learning, and a visual servoing controller, forming a paradigm that combines the advantages of data-driven policy and classic controller. The visual parsing uses stereo depth estimation and image segmentation with a visual foundation model to handle complex scenes. Experiments demonstrated autonomy in seven game-based skill training tasks on the da Vinci Research Kit, with a proof-of-concept study on haptic-assisted skill training as a practical application. Moreover, we conducted automation of five surgical assistive tasks with the Sentire surgical system on ex vivo animal tissues with various scenes, object sizes, instrument types, and illuminations. The learned policies were also validated in a live-animal trial for three tasks in dynamic in vivo surgical environments. We hope this open-source infrastructure, coupled with a general-purpose learning paradigm, will inspire and facilitate future research on embodied intelligence toward autonomous surgical robots.

The ability to recover tissue deformation from visual features is fundamental for many robotic surgery applications. This has been a long-standing research topic in computer vision, however, is still unsolved due to complex dynamics of soft tissues when being manipulated by surgical instruments. The ambiguous pixel correspondence caused by homogeneous texture makes achieving dense and accurate tissue tracking even more challenging. In this paper, we propose a novel self-supervised framework to recover tissue deformations from stereo surgical videos. Our approach integrates semantics, cross-frame motion flow, and long-range temporal dependencies to enable the recovered deformations to represent actual tissue dynamics. Moreover, we incorporate diffeomorphic mapping to regularize the warping field to be physically realistic. To comprehensively evaluate our method, we collected stereo surgical video clips containing three types of tissue manipulation (i.e., pushing, dissection and retraction) from two different types of surgeries (i.e., hemicolectomy and mesorectal excision). Our method has achieved impressive results in capturing deformation in 3D mesh, and generalized well across manipulations and surgeries. It also outperforms current state-of-the-art methods on non-rigid registration and optical flow estimation. To the best of our knowledge, this is the first work on self-supervised learning for dense tissue deformation modeling from stereo surgical videos. Our code will be released.

Laparoscopic hysterectomy is a common minimally invasive gynecologic surgery in which an assistant is required to manipulate the uterus during the procedure according to the verbal commands of the primary surgeon. However, uterus manipulation is a lengthy and laborious task, where human fatigue may lower operating safety. This paper presents a novel robot-enabled uterus manipulation system that provides accurate, tireless, and direct uterus manipulation. The system consists primarily of a 7-DoF robotic arm tailored for uterus manipulation in laparoscopic hysterectomy. The primary surgeon can directly control the robot using a footswitch pedal to move within the constraint of the Remote Center of Motion (RCM) via software and achieve the desired range of pitch and yaw motion. An additional rotational motion around the instrument axis is provided for twisting the uterus to enhance the exposure of the ligaments and peritoneum for resection. In addition, the robot arm consists of 7 compact modular actuators that provide the target payload for holding the uterus while maintaining a small footprint in a constrained operating environment. To facilitate operation, a passive mobile base is provided for supporting and positioning the robot. A 6-D force sensor is equipped that allows the assistant to quickly guide the robot into the work area and move the robot under RCM constraint using an admittance control approach. The uterus manipulation rod can be easily detached from the robot body by a quick pluggable mechanism for preoperative sterilization. A quick pluggable mechanism was proposed for easy separation and mounting of the uterus manipulation rod from the robot body for preoperative sterilization. Experiments were performed to measure the performance of the modular joints, RCM repeatability, and evaluate the feasibility of the robot in the simulated laparoscopic hysterectomy.

Purpose Automatic segmentation of surgical instruments in robot-assisted minimally invasive surgery plays a fundamental role in improving context awareness. In this work, we present an instance segmentation model based on refined Mask R-CNN for accurately segmenting the instruments as well as identifying their types. Methods We re-formulate the instrument segmentation task as an instance segmentation task. Then we optimize the Mask R-CNN with anchor optimization and improved Region Proposal Network for instrument segmentation. Moreover, we perform cross-dataset evaluation with different sampling strategies. Results We evaluate our model on a public dataset of the MICCAI 2017 Endoscopic Vision Challenge with two segmentation tasks, and both achieve new state-of-the-art performance. Besides, cross-dataset training improved the performance on both segmentation tasks compared with those tested on the public dataset. Conclusion Results demonstrate the effectiveness of the proposed instance segmentation network for surgical instruments segmentation. Cross-dataset evaluation shows our instance segmentation model presents certain cross-dataset generalization capability, and cross-dataset training can significantly improve the segmentation performance. Our empirical study also provides guidance on how to allocate the annotation cost for surgeons while labelling a new dataset in practice.

Compliant joints can provide safer physical human-robot interaction (pHRI). To effectively balance the response bandwidth and output impedance, variable stiffness joints have been studied by many researchers. Leaf spring based variable stiffness joint (LSVSJ) has the advantages of easy construction and large stiffness range. This paper presents the design and validation of a novel leaf spring based variable stiffness joint with reconfigurability (LSJR). Compared with most of the existing LSVSJ, this novel LSJR ensures the contact performance between the roller and the leaf spring, uses a novel effective length adjustment mechanism, and provides multiple stiffness ranges. The compliance of the LSJR is generated by leaf springs. The spring is bent by two rows of rollers integrated on an adjustable slider, where the distance between the rollers can be changed so as to achieve a tighter contact between the roller and the spring. The effective length of the spring can be adjusted by changing the position of the slider locating at the intersection between a straight rail and an arcuate rail. A reconfigurable number of leaf springs provide six different stiffness ranges (e.g., 138.55~396.08 Nm/rad for single leaf spring, and 384.56~1222.60 Nm/rad for a combination of three leaf springs). By integrating a high-resolution angle sensor, the LSJR can also measure the joint torque. Moreover, the hollow shaft structure of the joint is beneficial for passing cables through it. Experiments were performed to measure the stiffness of the LSJR under different configurations, validate its efficacy in unexpected collisions, and investigate its potential to be a one-degree-of-freedom (DOF) torque sensor.

Precise 3-D surface reconstruction plays an important role in automated manipulation, industrial inspection, robotics, and so on. In this article, we present a novel 3-D surface reconstruction framework for stereo vision systems assisted with structured light projection. In the framework, a multistep matching scheme is proposed to establish a reliable correspondence between image pairs with high computation efficiency and accuracy. The successive matching steps can find the most precise correspondence through a step-by-step filtering procedure. To further enhance the precision, a correspondence refinement algorithm is presented. Phase maps with different frequencies are utilized as the code words for the multistep matching due to their high encoding accuracy and robustness to noise. This method does not require phase unwrapping or projector calibration, which improves the reconstruction precision and simplifies the operation. Selection strategies for the number of matching steps, the pattern frequencies, and the matching threshold are proposed. Furthermore, various 3-D reconstruction experiments are conducted using the proposed framework. Comparative experiments verify the advantages of the proposed framework compared with existing 3-D reconstruction methods regarding the accuracy and precision. The adaptability to scenarios with different motion speeds is demonstrated. Robustness and limitations of the framework are also revealed by conducting experiments in challenging scenarios.

This paper addresses the problem of controlling a continuum manipulator (CM) in free or obstructed environments with no prior knowledge about the deformation behavior of the CM and the stiffness and geometry of the interacting obstructed environment. We propose a versatile data-driven priori-model-independent (PMI) control framework, in which various control paradigms (e.g. CM's position or shape control) can be defined based on the provided feedback. This optimal iterative algorithm learns the deformation behavior of the CM in interaction with an unknown environment, in real time, and then accomplishes the defined control objective. To evaluate the scalability of the proposed framework, we integrated two different CMs, designed for medical applications, with the da Vinci Research Kit (dVRK). The performance and learning capability of the framework was investigated in 11 sets of experiments including PMI position and shape control in free and unknown obstructed environments as well as during manipulation of an unknown deformable object. We also evaluated the performance of our algorithm in an ex-vivo experiment with a lamb heart.The theoretical and experimental results demonstrate the adaptivity, versatility, and accuracy of the proposed framework and, therefore, its suitability for a variety of applications involving continuum manipulators.

This paper aims to propose a 3D laparoscopic imaging system that can realize dense 3D reconstruction in real time. Based on the active stereo technique which yields high-density, accurate and robust 3D reconstruction by combining structured light and stereo vision, we design a laparoscopic system consisting of two image feedback channels and one pattern projection channel. Remote high-speed image acquisition and pattern generation lay the foundation for the real-time dense 3D surface reconstruction and enable the miniaturization of the laparoscopic probe. To enhance the reconstruction efficiency and accuracy, we propose a novel active stereo method by which the dense 3D point cloud is obtained using only five patterns, while most existing multiple-shot structured light techniques require 10-40 patterns. In our method, dual-frequency phase-shifting fringes are utilized to uniquely encode the pixels of the measured targets, and a dual-codeword matching scheme is developed to simplify the matching procedure and achieve high-precision reconstruction. Compared with the existing structured light techniques, the proposed method shows better real-time efficiency and accuracy in both quantitative and qualitative ways. Ex-vivo experiments demonstrate the robustness of the proposed method to different biological organs and the effectiveness to lesions and deformations of the organs. Feasibility of the proposed system for real-time dense 3D reconstruction is verified in dynamic experiments. According to the experimental results, the system acquires 3D point clouds with a speed of 12 frames per second. Each frame contains more than 40,000 points, and the average errors tested on standard objects are less than 0.2 mm. This paper provides a new real-time dense 3D reconstruction method for 3D laparoscopic imaging. The established prototype system has shown good performance in reconstructing surface of biological tissues.

Conventional robot hand-eye calibration methods are impractical for localizing robotic instruments in minimally-invasive surgeries under intra-corporeal workspace after pre-operative set-up. In this letter, we present a new approach to autonomously calibrate a robotic instrument relative to a monocular camera without recognizing calibration objects or salient features. The algorithm leverages interactive manipulation (IM) of the instrument for tracking its rigid-body motion behavior subject to the remote center-of-motion constraint. An adaptive controller is proposed to regulate the IM-induced instrument trajectory, using visual feedback, within a 3D plane which is observable from both the robot base and the camera. The eye-to-hand orientation and position are then computed via a dual-stage process allowing parameter estimation in low-dimensional spaces. The method does not require the exact knowledge of the instrument model or large-scale data collection. Results from simulations and experiments on the da Vinci Research Kit are demonstrated via a laparoscopy resembling set-up using the proposed framework.

Field of view (FOV) is important for minimally invasive surgeries. However, manual laparoscope manipulation is laborious and requires a lot of training. Assistants’ fatigue and misunderstanding during collaboration will also compromise the quality of FOV. To improve these drawbacks, we developed a new algorithm to automatically manipulate 2-D laparoscopes. The innovative aspect lies in the following ways: the algorithm designs adaptive laws for on-line estimation of depth information, which is difficult to calibrate accurately in advance; To follow moving tools and stabilize FOV around local operations at the same time, a region-based visual servoing method is proposed. We design a novel region factor to deal with regions of different shapes; Aside from the remote center of motion (RCM), the eye-hand coordination problems, especially misorientation, are also key issues for MIS. However, how to reduce the misorientation effects is still an open problem in the literature. We propose a new constraint called the intuitive virtual plane (IVP) constraint to minimize the misorientation; In contrast with the most related works assuming the optical axis and physical axis of a laparoscope are perfectly coincident to each other, our method can be extended to different kinds of laparoscopes without any assumptions on its optical and physical axis with null-space method to decouple image-based visual servoing (IBVS) and RCM-IVP constraints. Experiments are presented to validate the method. Moreover, user studies and interviews are conducted with medical professionals and inexperienced subjects, and the results show that our system is able to improve efficiency, eye-hand coordination, and safety for FOV control of laparoscopes.