Youjie Zhou

Despite significant advancements in the medical field, surgical robots still exhibit limited autonomy. Image guidance techniques are widely regarded as a key pathway toward achieving full automation in surgical robotics, and deep learning (DL) has further accelerated progress in this domain. However, to our knowledge, no existing study has systematically reviewed the key technologies involved. Therefore, this article provides a comprehensive analysis of the critical technologies underlying DL-based image guidance for surgical robots. First, we introduce medical image classification methods and commonly used datasets. The discussion then examines five preoperative technologies in image guidance, including image segmentation, registration and fusion, 3-D reconstruction, and surgical planning, with emphasis on their current research status and associated challenges. Next, we review four intraoperative critical technologies, including real-time registration and planning, surgical instrument tracking, and soft-tissue detection, while summarizing the challenges and limitations of DL applications in this domain. We also explore the potential applications of advanced technologies in image-guided surgical robotics, such as extended reality and foundation models. Furthermore, by consolidating preoperative, intraoperative, and advanced critical technologies, we propose a five-level framework for evaluating image-guidance capabilities. Finally, we outline research trends and future directions.

Visual SLAM is particularly challenging in environments affected by noise, varying lighting conditions, and darkness. Learning-based optical flow algorithms can leverage multiple modalities to address these challenges, but traditional optical flow-based visual SLAM approaches often require significant computational resources. To overcome this limitation, we propose FMF-SLAM, an efficient multimodal fusion SLAM method that utilizes fast Fourier transform (FFT) to enhance the algorithm efficiency. Specifically, we introduce a novel Fourier-based self-attention and cross-attention mechanism to extract features from RGB and depth signals. We further enhance the interaction of multimodal features by incorporating multi-scale knowledge distillation across modalities. We also demonstrate the practical feasibility of FMF-SLAM in real-world scenarios with real time performance by integrating it with a security robot by fusing with a global positioning module GNSS-RTK and global Bundle Adjustment. Our approach is validated using video sequences from TUM, TartanAir, and our real-world datasets, showcasing state-of-the-art performance under noisy, varying lighting, and dark conditions. Our code and datasets are available at https://github.com/youjie-zhou/FMF-SLAM.git .

This letter presents an investigation into the estimation of optical and scene flow using RGBD information in scenarios where the RGB modality is affected by noise or captured in dark environments. Existing methods typically rely solely on RGB images or fuse the modalities at later stages, which can result in lower accuracy when the RGB information is unreliable. To address this issue, we propose a novel deep neural network approach named FusionRAFT, which enables early-stage information fusion between sensor modalities (RGB and depth). Our approach incorporates self- and cross-attention layers at different network levels to construct informative features that leverage the strengths of both modalities. Through comparative experiments, we demonstrate that our approach outperforms recent methods in terms of performance on the synthetic dataset FlyingThings3D, as well as the generalization on the real-world dataset KITTI. We illustrate that our approach exhibits improved robustness in the presence of noise and low-lighting conditions that affect the RGB images.

We present a simple yet effective method to address loop closure detection in simultaneous localisation and mapping using local 3D deep descriptors (L3Ds). L3Ds are emerging compact representations of patches extracted from point clouds that are learnt from data using a deep learning algorithm. We propose a novel overlap measure for loop detection by computing the metric error between points that correspond to mutually-nearest-neighbour descriptors after registering the loop candidate point cloud by its estimated relative pose. This novel approach enables us to accurately detect loops and estimate six degrees-of-freedom poses in the case of small overlaps. We compare our L3D-based loop closure approach with recent approaches on LiDAR data and achieve state-of-the-art loop closure detection accuracy. Additionally, we embed our loop closure approach in RESLAM, a recent edge-based SLAM system, and perform the evaluation on real-world RGBD-TUM and synthetic ICL datasets. Our approach enables RESLAM to achieve a better localisation accuracy compared to its original loop closure strategy.

With the development of the times, robotics technology has gradually been no longer confined to the laboratory, but more integrated into the lives of the public. As a researcher in the field of intelligent science, one should contribute his own contribution to the development of robotics. The visual processing of robots is an important topic in today's science and technology circles, and my team and I have developed a strong interest in this field. For the specific realization of visual processing, we have selected a technology that is very close to the lives of ordinary people-SLAM mapping technology and application for related research. I will use the construction of ROS robot as a starting point to briefly explain how to develop and apply SLAM algorithms on the ROS platform. At the same time, I will explore the derivation of SLAM algorithms-the role and difference of Hector algorithm and Karto algorithm in building maps. By sharing some of my insights on SLAM technology, I believe it can provide some reference for subsequent research.

Wearable cameras, such as Google Glass and Go Pro, enable video data collection over larger areas and from different views. In this paper, we tackle a new problem of locating the co-interest person (CIP), i.e., the one who draws attention from most camera wearers, from temporally synchronized videos taken by multiple wearable cameras. Our basic idea is to exploit the motion patterns of people and use them to correlate the persons across different videos, instead of performing appearance-based matching as in traditional video co-segmentation/localization. This way, we can identify CIP even if a group of people with similar appearance are present in the view. More specifically, we detect a set of persons on each frame as the candidates of the CIP and then build a Conditional Random Field (CRF) model to select the one with consistent motion patterns in different videos and high spacial-temporal consistency in each video. We collect three sets of wearable-camera videos for testing the proposed algorithm. All the involved people have similar appearances in the collected videos and the experiments demonstrate the effectiveness of the proposed algorithm.