Xueying Sun

Scene graph generation (SGG) addresses the task of detecting objects in an image and predicting the relationships among them. Although prototype-based methods have recently achieved clear progress on long-tailed SGG, fine-grained low-frequency predicates remain difficult to recognize because their relation features often exhibit larger intra-class variation and more dispersed distributions, making them easily confused with semantically similar high-frequency coarse-grained predicates under a unified prototype-matching rule. To alleviate this issue, we propose a frequency-aware elastic prototype boundary learning framework, termed SGE-Net. Under fixed relation prototypes, the framework learns relation-category-specific boundary scales through explicit frequency compensation and frequency-adaptive virtual sampling, so that relation prediction can exploit not only prototype-center matching but also category-dependent decision-boundary information. During inference, we further introduce elastic boundary-aware distance calibration, enabling the boundary information learned during training to better distinguish relation categories that are easily confused under prototype matching. In addition, we combine visual and semantic features with dynamic gating to provide more reliable relation features for the above boundary learning. Experiments and analyses on Visual Genome and Open Images V6 demonstrate that the proposed method achieves consistent gains in both long-tailed relation prediction and overall evaluation metrics.

Panoptic scene graph generation (PSG) aims to simultaneously segment both foreground objects and background regions while predicting object relations for fine-grained scene modeling. Despite significant progress in panoptic scene understanding, current PSG methods face challenging problems: relation prediction often only relies on visual representations and is hindered by imbalanced relation category distributions. Accordingly, we propose IntegraPSG, a single-stage framework that integrates large language model (LLM) guidance with multimodal feature fusion. IntegraPSG introduces a multimodal sparse relation prediction network that efficiently integrates visual, linguistic, and depth cues to identify subject–object pairs most likely to form relations, enhancing the screening of subject–object pairs and filtering dense candidates into sparse, effective pairs. To alleviate the long-tail distribution problem of relations, we design a language-guided multimodal relation decoder where LLM is utilized to generate language descriptions for relation triplets, which are cross-modally attended with vision pair features. This design enables more accurate relation predictions for sparse subject–object pairs and effectively improves discriminative capability for rare relations. Experimental results show that IntegraPSG achieves steady and strong performance on the PSG dataset, especially with the R@100, mR@100, and mean reaching 38.7%, 28.6%, and 30.0%, respectively, indicating strong overall results and supporting the validity of the proposed method.

Hyperspectral images are rich in spectral and spatial information. However, their high dimensionality and complexity pose significant challenges for effective feature extraction. Specifically, the performance of existing models for hyperspectral image (HSI) classification remains constrained by spectral redundancy among adjacent bands, misclassification at object boundaries, and significant noise in hyperspectral data. To address these challenges, we propose WTCMC—a novel hyperspectral image classification network based on wavelet transform combining Mamba and convolutional neural networks. To establish robust shallow spatial–spectral relationships, we introduce a shallow feature extraction module (SFE) at the initial stage of the network. To enable the comprehensive and efficient capture of both spectral and spatial characteristics, our architecture incorporates a low-frequency spectral Mamba module (LFSM) and a high-frequency multi-scale convolution module (HFMC). The wavelet transform suppresses noise for LFSM and enhances fine spatial and contour features for HFMC. Furthermore, we devise a spectral–spatial complementary fusion module (SCF) that selectively preserves the most discriminative spectral and spatial features. Experimental results demonstrate that the proposed WTCMC network attains overall accuracies (OA) of 98.94%, 98.67%, and 97.50% on the Pavia University (PU), Botswana (BS), and Indian Pines (IP) datasets, respectively, outperforming the compared state-of-the-art methods.

In recent years, graph convolutional neural networks (GCNs) and convolutional neural networks (CNNs) have made significant strides in hyperspectral image (HSI) classification. However, existing models often encounter information redundancy and feature mismatch during feature fusion, and they struggle with small-scale refined features. To address these issues, we propose DCG-Net, an innovative classification network integrating CNN and GCN architectures. Our approach includes the development of a double-branch expanding network (E-Net) to enhance spectral features and efficiently extract high-level features. Additionally, we incorporate a GCN with an attention mechanism to facilitate the integration of multi-space scale superpixel-level and pixel-level features. To further improve feature fusion, we introduce a feature aggregation module (FAM) that adaptively learns channel features, enhancing classification robustness and accuracy. Comprehensive experiments on three widely used datasets show that DCG-Net achieves superior classification results compared to other state-of-the-art methods.

Ensuring safe and efficient navigation in crowded environments is a critical goal for assistive robots. Recent studies have emphasized the potential of deep reinforcement learning techniques to enhance robots’ navigation capabilities in the presence of crowds. However, current deep reinforcement learning methods often face the challenge of robots freezing as crowd density increases. To address this issue, a novel risk-aware deep reinforcement learning approach is proposed in this paper. The proposed method integrates a risk function to assess the probability of collision between the robot and pedestrians, enabling the robot to proactively prioritize pedestrians with a higher risk of collision. Furthermore, the model dynamically adjusts the fusion strategy of learning-based and risk-aware-based features, thereby improving the robustness of robot navigation. Evaluations were conducted to determine the effectiveness of the proposed method in both low- and high-crowd density settings. The results exhibited remarkable navigation success rates of 98.0% and 93.2% in environments with 10 and 20 pedestrians, respectively. These findings emphasize the robust performance of the proposed method in successfully navigating through crowded spaces. Additionally, the approach achieves navigation times comparable to those of state-of-the-art methods, confirming its efficiency in accomplishing navigation tasks. The generalization capability of the method was also rigorously assessed by subjecting it to testing in crowd environments exceeding the training density. Notably, the proposed method attains an impressive navigation success rate of 90.0% in 25-person environments, surpassing the performance of existing approaches and establishing itself as a state-of-the-art solution. This result highlights the versatility and effectiveness of the proposed method in adapting to various crowd densities and further reinforces its applicability in real-world scenarios.

Recently, applying the utilization of RGB-D data for robot perception tasks has garnered significant attention in domains like robotics and autonomous driving. However, a prominent challenge in this field lies in the substantial impact of feature robustness on both segmentation and pose estimation tasks. To tackle this challenge, we proposed a pioneering two-stage hybrid Convolutional Neural Network (CNN) architecture, which connects segmentation and pose estimation in tandem. Specifically, we developed Cross-Modal (CM) and Cross-Layer (CL) modules to exploit the complementary information from RGB and depth modalities, as well as the hierarchical features from diverse layers of the network. The CM and CL integration strategy significantly enhanced the segmentation accuracy by effectively capturing spatial and contextual information. Furthermore, we introduced the Convolutional Block Attention Module (CBAM), which dynamically recalibrated the feature maps, enabling the network to focus on informative regions and channels, thereby enhancing the overall performance of the pose estimation task. We conducted extensive experiments on benchmark datasets to evaluate the proposed method and achieved exceptional target pose estimation results, with an average accuracy of 94.5% using the ADD-S AUC metric and 97.6% of ADD-S smaller than 2 cm. These results demonstrate the superior performance of our proposed method.

Accurately detecting suitable grasp areas for unknown objects through visual information remains a challenging task. Drawing inspiration from the success of the Vision Transformer in vision detection, the hybrid Transformer-CNN architecture for robotic grasp detection, known as HTC-Grasp, is developed to improve the accuracy of grasping unknown objects. The architecture employs an external attention-based hierarchical Transformer as an encoder to effectively capture global context and correlation features across the entire dataset. Furthermore, a channel-wise attention-based CNN decoder is presented to adaptively adjust the weight of the channels in the approach, resulting in more efficient feature aggregation. The proposed method is validated on the Cornell and the Jacquard dataset, achieving an image-wise detection accuracy of 98.3% and 95.8% on each dataset, respectively. Additionally, the object-wise detection accuracy of 96.9% and 92.4% on the same datasets are achieved based on this method. A physical experiment is also performed using the Elite 6Dof robot, with a grasping accuracy rate of 93.3%, demonstrating the proposed method’s ability to grasp unknown objects in real scenarios. The results of this study indicate that the proposed method outperforms other state-of-the-art methods.

In the field of vision-based robot grasping, effectively leveraging RGB and depth information to accurately determine the position and pose of a target is a critical issue. To address this challenge, we proposed a tri-stream cross-modal fusion architecture for 2-DoF visual grasp detection. This architecture facilitates the interaction of RGB and depth bilateral information and was designed to efficiently aggregate multiscale information. Our novel modal interaction module (MIM) with a spatial-wise cross-attention algorithm adaptively captures cross-modal feature information. Meanwhile, the channel interaction modules (CIM) further enhance the aggregation of different modal streams. In addition, we efficiently aggregated global multiscale information through a hierarchical structure with skipping connections. To evaluate the performance of our proposed method, we conducted validation experiments on standard public datasets and real robot grasping experiments. We achieved image-wise detection accuracy of 99.4% and 96.7% on Cornell and Jacquard datasets, respectively. The object-wise detection accuracy reached 97.8% and 94.6% on the same datasets. Furthermore, physical experiments using the 6-DoF Elite robot demonstrated a success rate of 94.5%. These experiments highlight the superior accuracy of our proposed method.

We introduce a novel hybrid Transformer-CNN architecture for robotic grasp detection, designed to enhance the accuracy of grasping unknown objects. Our proposed architecture has two key designs. Firstly, we develop a hierarchical transformer as the encoder, incorporating the external attention to effectively capture the correlation features across the data. Secondly, the decoder is constructed with cross-layer connections to efficiently fuse multi-scale features. Channel attention is introduced in the decoder to model the correlation between channels and to adaptively recalibrate the channel correlation feature response, thereby increasing the weight of the effective channels. Our method is evaluated on the Cornell and Jacquard public datasets, achieving an image-wise detection accuracy of 98.3% and 95.8% on each dataset, respectively. Additionally, we achieve object-wise detection accuracy of 96.9% and 92.4% on the same datasets. A physical experiment is also performed using the Elite 6Dof robot, with a grasping accuracy rate of 93.3%, demonstrating the proposed method's ability to grasp unknown objects in real-world scenarios. The results of this study show that our proposed method outperforms other state-of-the-art methods.