Qixin Cao

Robust and efficient contactless human–machine interaction is critical for integrated sensor systems in clinical settings, demanding low-power solutions adaptable to edge computing platforms. This paper presents a real-time hand gesture recognition system using a low-power Frequency-Modulated Continuous Wave (FMCW) radar sensor, featuring a novel Multiple Feature Fusion (MFF) framework optimized for deployment on edge devices. The proposed system integrates velocity profiles, angular variations, and spatial-temporal features through a dual-stage processing architecture: an adaptive energy thresholding detector segments gestures, followed by an attention-enhanced neural classifier. Innovations include dynamic clutter suppression and multi-path cancellation optimized for complex clinical environments. Experimental validation demonstrates high performance, achieving 98% detection recall and 93.87% classification accuracy under LOSO cross-validation. On embedded hardware, the system processes at 28 FPS, showing higher robustness against environmental noise and lower computational overhead compared with existing methods. This low-power, edge-based solution is highly suitable for applications like sterile medical control and patient monitoring, advancing contactless interaction in healthcare by addressing efficiency and robustness challenges in radar sensing for edge computing.

The 6D pose estimation for metal parts is essential in industrial robotic applications. Although the 6D pose estimation methods that rely on object-specific training have gained extensive concern, these methods can’t generalize to novel objects. Recent novel object pose estimation methods are solving this issue using task-specific fine-tuned CNNs for deep template matching. However, these methods require expensive training procedures and don’t consider pose distribution of rendered templates. Recently, foundation models show strong representation learning ability, and can encode both the high spatial information as well as semantic information. In this study, we present a metal parts’ zero-shot 6D pose estimation method by foundation model without re-training, and incorporate prior information of metal parts’ poses to generate rendered templates that align with the pose distribution in real world. DINOv2, a recent vision foundation model with impressive generalization capabilities, is employed for matching rendered templates against query images of metal parts. Additionally, we introduce the optimal transport as a similarity metric. Then, the Gluestick is utilized to establish local keypoint correspondences, which enable deriving geometric correspondences and are used for estimating the metal part’s 6D pose with PnP/RANSAC. Given an industrial scenario, we first estimate metal parts’ pose, select and save high-confidence pose results to pose buffer, which is used to update the templates. The experiments on MP6D and ROBI datasets showcase that the proposed method has better performance than MegaPose. We also conduct real world experiments, which demonstrate the robustness of the proposed method. Code is available at https://github.com/sunhan1997/IndusPose.

Patients with infectious diseases often exhibit severe respiratory distress and breathing difficulties in isolation wards. Tracheal intubation is performed manually to secure the airway by health care providers wearing personal protective equipment to prevent infection spread and getting infected. This paper presents the navigation and preoperative positioning of an autonomous mobile intubation robot capable of independently moving to the patient’s bed, positioning the intubation device inside the mouth while the operator completes the intubation procedure remotely, and the robot exits the location independently. The mobile base navigation module includes mapping, localization and path planning using the R3 live multi-sensor fusion algorithm utilizing sensor imputs from lidar, inertia measurement unit (IMU) and RGB-D camera, YOLOv5s object detection and avoidance algorithm and improved A* global path planning algorithm together with time elastic band (TEB) local path planning algorithm respectively. The preoperative positioning module based on facial pose estimation using the retinaface algorithm combined with 6DRepNet to obtain facial pose. This is validated in simulation and physical experiments. The system’s navigation time and positioning accuracy compared with related indicators of endotracheal intubation surgery obtained significantly improved results.

Patients with infectious diseases often exhibit severe respiratory distress and breathing difficulties in isolation wards. Tracheal intubation is performed manually to secure the airway by health care providers wearing personal protective equipment to prevent infection spread and getting infected. This paper presents the navigation and preoperative positioning of an autonomous mobile intuba-tion robot capable of independently moving to the patient’s bed, positioning the intubation device inside the mouth while the operator completes the intubation procedure remotely, and the robot exits the location independently. The mobile base navigation module includes mapping, localization and path planning using the R3 live multi-sensor fusion algorithm utilizing sensor imputs from lidar, inertia measurement unit (IMU) and RGB-D camera, YOLOv5s object detection and avoidance algorithm and improved A* global path planning algorithm together with time elastic band (TEB) local path planning algorithm respectively. The preoperative positioning module based on facial pose estimation using retinaface algorithm combined with 6DRepNet to obtain facial pose. This is validated in simulation and physical experiments. The system’s navigation time and positioning accuracy compared with related indicators of endotracheal intubation surgery obtained significantly improved results.

Existing visual simultaneous localization and mapping (SLAM) systems struggle with loop closure under significant viewpoint variations, such as revisiting the same place orthogonally or oppositely. This limitation primarily stems from the lack of viewpoint invariance in both macroscopic place definition and microscopic feature description. Based on the crucial insight that geometric information is more viewpoint-invariant than visual information, we create map segments via leveraging the position and coplanarity distribution of map elements within the map constructed by monocular SLAM to overcome the limitations of frame-based place definition, and extract perspective-invariant ORB (PRIOR) features via accounting for local surface perspective distortions to enhance perspective invariance without the need for costly perspective invariance estimation or additional depth data. We further utilize map segments and PRIOR features to hierarchically detect and correct loop closures with a coarse-to-fine geometric consistency check. We integrate our novelties into the prevalent SLAM framework, thereby proposing PRIOR-SLAM, which achieves state-of-the-art performance in feature matching and retrieval, visual place recognition, and loop closure under large viewpoint changes.

The open-loop grasp planner, which relies on vision, is prone to failure caused by calibration errors, visual occlusions, and other factors. Additionally, it cannot adapt the grasp pose and gripping force in real-time, thereby increasing the risk of potential damage to unidentified objects. This work presents a multimodal regrasp control framework based on deep reinforcement learning (RL). Given a coarse initial grasp pose, the proposed regrasping policy efficiently optimizes grasp pose and gripping force by deeply fusing visual and high-resolution tactile data in a closed-loop fashion. To enhance the sample efficiency and generalization capability of the RL algorithm, this work leverages self-supervision to pre-train a visual-tactile representation model, which serves as a feature extraction network during RL policy training. The RL policy is trained purely in simulation and successfully deployed to a real-world environment via domain adaptation and domain randomization techniques. Extensive experimental results in simulation and real-world environments indicate that the robot guided by the regrasping policy is able to achieve gentle grasping of unknown objects with high success rates. Finally, the comparison results with the state-of-the-art algorithm also demonstrate the superiority of our algorithm.

The 6D pose estimation for metal parts is essential in industrial robotic applications. The color homogeneity, texture-less and light-reflecting properties of metal parts raise great challenges. Current 6D pose estimation methods have gained extensive concern using CNNs. However, these CNN-based methods lack Transformer's ability to focus on extracting low-frequency features and long-range context information. In the study, we explore taking full advantage of CNN and Transformer from a frequency-domain perspective to enhance the performance of metal parts' 6D pose estimation. Specifically, we propose a frequency-guided CNN-Transformer fusion 6D pose estimation network (FGCT6D). First, we construct a novel pixel attention residual module to improve the high-frequency attention of CNN. Then, we design a dual-branch CNN-Transformer encoder: the Swin-Transformer extracts global information and low-frequency features, and the CNN captures local information and high-frequency features. Second, the frequency-guided feature fusion module is proposed to fuse the extracted multi-spectral features. Third, to maximize the utilization of the rich frequency-domain feature representation, we propose a feature fusion decoder with Conv-MSA modules. Additionally, we leverage optimal transport theory, treating dense correspondences as spatial probability distributions, and design the optimal transport loss function. Experiments show that our method can extract rich frequency-domain features, and achieve competitive performance on the MP6D and LINEMOD datasets.

Industrial bin picking is a challenging task as lots of highly reflective and textureless objects are stacked. Previous methods generally predict gripper configurations or 6D object pose by expensive industrial 3D camera, ignoring the role of consumer-grade RGB-D cameras in this task. For highly reflective objects, depth maps of consumer-grade RGB-D cameras are generally of a lower quality with much stronger noise, making it challenging to detect robust grasp poses in stacked industrial scenarios. To address these challenges, we propose a novel grasp detection framework including a synthetic data generation pipeline, a novel local grasp detection method, and a global search algorithm. Specifically, in the training phase, we utilize the Pybullet and OpenGL to generate synthetic data by approximate 3D models. We introduce a novel grasp representation and propose MixStyle-ResUnet to predict and achieve sim2real transfer for this representation. In the testing phase, the Object Singleness Metric (OSM) is proposed for the global search block to evaluate the graspable probability. In addition, the RGB image enhanced by edge feature is utilized as the input of the network, which can bridge the domain shift. Extensive experiments in real scenes demonstrate that our method can achieve competitive results in grasping disordered metal parts. In over 2400 robotic grasp trials, our method achieves an average success rate of 91.1% in dense stack scenarios. We also confirm that the proposed method can be applied to unseen objects which are not included in the training dataset. Code and videos are available at https://github.com/sunhan1997/IndusGrasp.

High-precision robotic grasping is necessary for extensive grasping applications in the future. Most previous grasp detection methods fail to pay enough attention to learn grasp-related features and the detection accuracy is limited. In this letter, a novel attention-augmented grasp detection network (AAGDN) is presented to generate accurate grasp poses for unknown objects. The proposed AAGDN has three elaborate designs making it achieve higher accuracy than existing methods. First, we construct a coordinate attention residual module to extract positional information and improve the spatial sensitivity of features. Then, we propose an effective feature fusion module to bridge the resolution and semantic gaps of different-level features and obtain efficient feature representations. Lastly, a feature augmentation pyramid module is developed to enhance grasp-related features as needed and reduce the loss of information. Extensive experiments on three public datasets and various real-world scenes prove that the proposed AAGDN achieves better performance than current methods. Our model obtains the state-of-the-art 99.3% and 96.2% grasp detection accuracy on the Cornell and Jacquard dataset, respectively. Moreover, in physical grasping experiments, the AAGDN attains the 94.6% success rate for unseen objects in cluttered scenes, which further demonstrates the accuracy and robustness of our method to grasp novel objects.