Yang Mo

The generalization ability of visual reinforcement learning, which allows the policy trained in the source domain to guide agents in similar unknown target environments, is one of the cores applied to visual navigation and autonomous driving. Recently, methods such as data augmentation techniques, self-supervised learning methods, and the generative adversarial network were employed to enhance the generalization capability of policy neural networks in visual reinforcement learning. However, current state-of-the-art methods, after utilizing domain-general latent features to train the RL policy, result in the loss of certain state-specific features, leading to diminished policy performance following generalization. To tackle these challenges, we designed a technical framework called self-expert imitation with purifying latent features, which enables the trained policy to effectively guide agents in scenarios similar to the training environment, without compromising the performance of the policy-guided agent in task completion. Additionally, a novel method was developed for separating domain-general and domain-specific latent vectors based on a variational autoencoder, enabling the domain-general component to exhibit strong and stable zero-shot generalization performance in unseen visually similar domains. Extensive experiments on the CarRacing game demonstrated that our approach achieves strong and stable generalization performance in unseen environments, without compromising the performance of the policy in guiding agents to complete tasks.

Gesture recognition constitutes a core modality for human–robot interaction (HRI). Current studies primarily address short-distance scenarios, whereas long-distance outdoor scenarios remain significantly underexplored. The principal technical challenges arise from three key challenges: First, the predominant separated spatio-temporal processing approach in current network architectures and attention mechanisms. 1) introduces redundant structural complexity and computational demands, severely constraining unmanned aerial vehicle (UAV) platform implementation. 2) Inherently limits the model’s ability to learn weakly-correlated spatio-temporal relationships. 3) Additionally, existing HRI systems demonstrate inherent deficiencies in accurate hand localization and skeleton estimation under long-distance conditions. To address these issues, we propose a lightweight spatio-temporal skeleton (STS) attention transformer for long-distance gesture recognition in UAV control. First, we designed and implemented a lightweight STS Transformer to unify spatial-temporal feature processing while maintaining structural simplicity. Second, we developed an STS attention employing coupled-decoupled operators on skeleton sequences to extract weakly-correlated features. Subsequently, we developed a two-stage RGB+D framework for hand detection and skeleton estimation, achieving capability at extended ranges up to 5 m under UAV onboard camera view and 10 m in third-party camera view. Finally, extensive experiments on two datasets and real-world experiments were conducted to demonstrate the effectiveness of our network in skeleton-based gesture recognition, showing a clear advantage over the state-of-the-art (SOTA) methods.

Image stitching plays a pivotal role in the quality inspection of printed circuit boards (PCBs). However, this process encounters significant challenges owing to the limited overlapping regions between the stitched images. Existing methods aim to alleviate the influence of features from non-overlapping regions on network performance by regularizing the loss function for these regions. Nevertheless, this approach incompletely eradicates interference from non-overlapping regions. Moreover, PCB images often comprise numerous homogeneous regions, where indistinguishable semantic information can result in erroneous alignments between feature points. In this paper, we propose a novel framework from the perspective of learning disentangled representations. Through an image reconstruction task, we disentangle the matching features of overlapping regions from the interfering features of non-overlapping regions, thereby attaining purified matching features and eliminating non-contributory features. Specifically, we partition the encoded features of the image into shared matching features from overlapping regions and interfering features from non-overlapping regions. By exchanging the shared features between image pairs to reconstruct the feature representations, we utilize these reassembled features to reconstruct the input image pairs. Furthermore, to address the ambiguity of homogeneous features in overlapping regions, we devise a Frequency Contextual Correlation Module (FCCM). The frequency convolution kernels in this module compute correlations between pixel blocks rather than individual pixels. Consequently, FCCM adeptly captures contextual semantic information between feature maps, resolving ambiguities induced by feature points in homogeneous regions. Extensive experimentation demonstrates that our approach effectively segregates matching alignment features from the overall image features while mitigating the interference of homogeneous region features. This contribution enhances alignment accuracy and achieves a 100% stitching success rate for PCBs, satisfying industrial requirements.

Stereo matching is of great importance in robot operation, autonomous driving and virtual reality. Large textureless regions and depth discontinuity regions are still the error-prone regions of stereo matching tasks. Traditional correlation-based volumes only measure the feature similarity within the same channel of the feature maps, resulting in insufficient feature similarity learning between different channels, which leads to poor performance of stereo networks in large textureless regions with high feature similarity requirements. To address the problems in large textureless regions, we propose the cross-correlation based cost volume construction which adequately learn the feature similarity in different channels of the feature maps. To address the problems in depth discontinuity regions and other gradient sensitive regions, we propose the differentiable least square aggregation module which can sufficiently utilize the gradient information and enhance the aggregation ability of the cost aggregation network for gradient features. Extensive experiments show that the proposed method solves the problems effectively in the above difficult regions and achieves competitive performance on Scene Flow dataset, KITTI 2012 dataset and KITTI 2015 dataset.

The core of multi-robot collision avoidance is to guide robots to avoid collisions with other robots and obstacles in a dynamic multi-robot environment, which has recently gained increasing interest among the main challenges of robotics. However, the current multi-robot navigation policy neural network exhibits weak position encoding capabilities for spatial environmental features in mapping environment states and robot actions, as well as an inability to recurrently infer information on dynamic environmental features in the temporal dimension, leading to insufficient safety and effectiveness in guiding robot motion. In this paper, we propose a novel spatial-temporal RetNet (STR) that encodes reciprocal collision avoidance states between robots in both spatial and temporal dimensions, aiming to enhance the safety and effectiveness of the policy neural network in guiding robots to accomplish specified tasks. The spatial state encoder module is developed based on parallel RetNet structure, which enhances the ability of the neural network in multi-robot navigation policies to extract reciprocal collision avoidance states between robots in spatial dimensions and overcomes the weak position encoding capability of advanced transformer-based multi-robot navigation policy neural networks. A temporal state encoder is designed by introducing the recurrent RetNet structure. This enhances the multi-robot navigation policy neural network’s ability to encode features in the temporal dimension of multi-robot movements and overcomes the transformer-based multi-robot navigation policy neural network’s inability to recurrently infer information in the time dimension. Simulation experiments were designed to demonstrate that the safety and effectiveness of our proposed method outperform the previous state-of-the-art approaches in guiding the robot to complete the task. Physical experiments illustrate that our policy can be effectively applied to real-world systems Note to Practitioners —Multi-robot navigation has a wide range of real-world applications, such as multi-robot formation flying for search and rescue, autonomous warehouse operations, and robots navigating through human crowds. This paper introduces a novel Spatial-Temporal RetNet (STR) framework aimed at enhancing safety and effectiveness in multi-robot collision avoidance. STR addresses the limitations of existing methods by improving the neural network’s ability to extract reciprocal collision avoidance states in both spatial and temporal dimensions. The spatial state encoder strengthens the extraction of spatial features, while the temporal state encoder improves the handling of time-dependent information. Simulation and physical experiments demonstrate that STR enhances robot navigation in dynamic environments, making it suitable for real-world applications such as multi-robot coordination.

For the safety of autonomous driving systems, recognizing the gestures of cyclists and controllers on traffic roads is crucial. However, general action recognition methods are unsuitable for traffic gestures. This is because, compared to general actions, traffic gestures exhibit distinct characteristics, namely: (1) short action duration; (2) atypical action posture; and (3) small dataset sizes . We propose a novel two-stream graph convolutional network (2S-GCN) to address these challenges. Firstly, we introduced a motion-guided module connecting the motion and joint stream. By leveraging the motion stream’s spatial expression capability, we guided the joint stream’s learning in spatial dimensions to enhance the network’s performance to short action durations. Secondly, we utilized a simplified skeleton topology that exclusively captures the upper body during data preprocessing, thereby preserving essential information while eliminating redundancy. This approach enhances the network’s adaptability to atypical action postures. Thirdly, we use a GCN embedded with a multi-channel attention module as the backbone, which is particularly suitable for small datasets, and conduct extensive experiments on two datasets. The experimental results demonstrate that our network can accurately recognize traffic gestures and has significant advantages over state-of-theart methods.

Depth estimation is one of the fundamental tasks of computer vision. Stereo matching is the most critical step to obtain the accurate depth information through stereo vision. At present, thin structure regions, depth discontinuity regions, and large textureless regions are still the difficult issues for stereo matching. To address the blur in thin structure regions and the dilation in depth discontinuity regions, the contextual structure enhancing module is proposed to enhance the extraction ability for local contextual features of the feature extraction network. To reduce the matching ambiguity in large textureless regions, the semantic feature enhancing module is proposed to enhance the aggregation ability for semantic features of the cost aggregation network. Extensive experiment results show that the proposed stereo network perform well in thin structure regions, depth discontinuity regions and large textureless regions and has achieved excellent performance on Scene Flow datasets, KITTI 2012 datasets, KITTI 2015 datasets and Middlebury datasets.

For autonomous driving systems, it is crucial to recognize the actions and gestures of traffic conductors and cyclists on the road to ensure safety. However, traffic gesture recognition is more challenging than action recognition in general scenarios due to the differences in action posture and sample composition between traffic gesture datasets and general action datasets. Therefore, general action recognition methods cannot identify traffic gestures well. To overcome these problems, we propose a novel motion-guided graph convolutional transformer (MG-GCT) for traffic gesture recognition. Firstly, we proposed a two-stream network to fully utilize joint data and motion data for action recognition. Secondly, we designed and implemented a motion-guided module between two streams, which leverages the powerful spatial representation ability of the motion data to guide the learning of the joint data stream in the spatial dimension. Thirdly, we implemented a temporal transformer network to process the temporal features of the skeleton. Finally, we conducted extensive experiments on two public datasets and one dataset presented by us to demonstrate the effectiveness of our network in traffic gesture recognition, which has a significant advantage over the state-of-the-art methods.