Rihui Song

Autonomous surgery involves surgical tasks performed by a robot with minimal or no human involvement. Thanks to its precise automation, surgical robotics offers significant benefits in enhancing the consistency, safety, and quality of procedures, driving its growing popularity. However, ensuring the safety of autonomous surgical robotic systems remains a significant challenge. To address this, we propose a simulation-based validation method to detect potential safety issues in the software of surgical robotic systems, complemented by a digital twin to estimate the gap between simulation and reality. The validation framework consists of a test case generator and a monitor for validating properties and evaluating the performance of the robotic system during test execution. Using a robotic arm for needle insertion as a case study, we present a systematic test case generation method that ensures effective coverage measurement for a three-dimensional, irregular model. Since no simulation can perfectly replicate reality due to differences in sensing and actuation, the digital twin bridges the gap between simulation and the physical robotic arm. This integration enables us to assess the discrepancy between virtual simulations and real-world operations by verifying whether the data from the simulation accurately predicts real-world outcomes. Through extensive experimentation, we identified several flaws in the robotic software. Co-simulation within the digital twin framework has highlighted these discrepancies that should be considered.

Dense depth estimation in endoscopic images can provide surgeons with important information for performing accurate minimally invasive surgeries. However, it is difficult to estimate the absolute depth of the scene based on monocular endoscope. Depth values in endoscopic images change drastically during the operation, which make it hard to estimate them with a fixed baseline. In this paper, we propose a depth estimation scheme with multiple baselines. The monocular endoscope is moved horizontally by a robotic endoscope holder to generate stereo images. A pixel-level depth map fusion algorithm is designed to combine depth values estimated with different baselines. Experimental results show that the proposed method improves the accuracy of depth estimation and the visual quality of depth maps.

Depth information is useful to surgeons and surgical assistance systems. However, it is a challenging task to estimate the depth of various surgical scenes based on a monocular endoscope. We propose a depth estimation approach for a monocular endoscope with a stereo matching algorithm. The monocular endoscope is moved horizontally by a robotic endoscope holder to simulate a stereo vision system. The main challenge is how to obtain a proper baseline for better depth information generation as the depth range of a surgical scene is unknown beforehand. We design a baseline evaluation and selection algorithm to search for suitable baselines for surgical scenes with different depth ranges. Experimental results show that our approach improves the average accuracy of different depth scenarios by 10.8% when the error range is 2mm.

Point clouds from 3-D light detection and ranging (LiDAR) are useful for roadside units (RSU) applications in intelligent transportation system (ITS). High-density LiDAR products are restricted by high cost while the low-density and cheap ones are usually insufficient to perceive. Event-based cameras react to the changes in light intensity and output dense event streams consisting of triggered pixels. Unfortunately, there currently lacks depth information for event cameras. In order to address these problems, this article presents an approach to enhance sparse 3-D LiDAR point clouds with event pixels from an event-based camera. In our approach, the depth values of event pixels are estimated based on the distribution models which they belong to. The distribution models of event pixels are determined by the spatial information of the neighboring LiDAR points in a structural manner which we called the physical structure. To verify our approach, we conduct several real-world experiments about RSU applications in ITS. Results demonstrate that our approach can effectively improve 3-D point clouds density. The average accuracy of 3-D and 2-D vehicle detection increase by a factor of 14.6 and 8.8, respectively.