XJXingshuo Jing

Xingshuo Jing

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Xingshuo Jing has a total of 16 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • CAGT: Sim-to-Real Depth Completion With Interactive Embedding Aggregation and Geometry Awareness for Transparent Objects

    by Xingshuo Jing, Kun Qian, Markus Vincze

    Robust depth completion of transparent objects would be beneficial for industrial automation such as vision-based robotic grasping and manipulation. However, although some methods try to learn a compact intra-layer feature representation with the boost of the attention mechanism or the vision Transformer, they ignore the neglected corner regions and sparse geometry information that are important for accurate depth completion. To tackle these issues, we propose a novel sim-to-real transferable model, named CAGT, with interactive embedding aggregation and geometry awareness to reconstruct severely sparse depth maps of transparent objects in this paper. We design a Depth-clue Interaction Aggregation Module (DIAM) to enhance the Transformer’s ability to extract boundary corner features and thus supplement depth clues. Then, we propose a Geometric Information Augmentation Module (GIAM) to fuse the geometry-aware feature containing shape and surface details. Moreover, we introduce a contrastive learning mechanism to facilitate the sim-to-real generalization of the completion model. Extensive experiment results on two challenging datasets, ClearGrasp and TransCG, demonstrate that our proposed CAGT can obtain superior performance over the state-of-the-art methods. We also demonstrate that CAGT can improve the grasp accuracy of transparent objects by a robotic grasping generalization experiment. The code and supplementary video will be available at: https://github.com/xingshuojing/CAGT.


  • Open Access

    Reducing Cross-Sensor Domain Gaps in Tactile Sensing via Few-Sample-Driven Style-to-Content Unsupervised Domain Adaptation

    by Xingshuo Jing, Kun Qian

    Transferring knowledge learned from standard GelSight sensors to other visuotactile sensors is appealing for reducing data collection and annotation. However, such cross-sensor transfer is challenging due to the differences between sensors in internal light sources, imaging effects, and elastomer properties. By understanding the data collected from each type of visuotactile sensors as domains, we propose a few-sample-driven style-to-content unsupervised domain adaptation method to reduce cross-sensor domain gaps. We first propose a Global and Local Aggregation Bottleneck (GLAB) layer to compress features extracted by an encoder, enabling the extraction of features containing key information and facilitating unlabeled few-sample-driven learning. We introduce a Fourier-style transformation (FST) module and a prototype-constrained learning loss to promote global conditional domain-adversarial adaptation, bridging style-level gaps. We also propose a high-confidence guided teacher–student network, utilizing a self-distillation mechanism to further reduce content-level gaps between the two domains. Experiments on three cross-sensor domain adaptation and real-world robotic cross-sensor shape recognition tasks demonstrate that our method outperforms state-of-the-art approaches, particularly achieving 89.8% accuracy on the DIGIT recognition dataset.


  • Reducing Sim-to-Real Domain Gaps of Visual Sensors for Learning Environment-Constrained Visuomotor Policy

    by Xingshuo Jing, Kun Qian, Boyi Duan

    Simulation engines enable safe training of robotic skills, but domain gaps between simulated and real sensors hinder deployment. However, existing pixel-level adaptation methods focus on visual realism of generating images over task-specific learning, causing texture leakage and elimination. In this paper, we introduce LCTGAN, an novel unsupervised domain transfer network, with a correlative attention mechanism and a mask-level Q-value mapping consistency to enhance task-awareness and bridge the domain gap of visual sensors in pixel-level perceptual manipulations. We also propose a Q-learning-based visuomotor policy to handle cluttered scenarios where objects may lack directly graspable configurations, which learns the synergies of three actions while considering environmental constraints. We further integrate LCTGAN into the learned policy to facilitate zero-shot sim-to-real policy transfer. Extensive experimental results validate the zero-shot sim-to-real generalization of our proposed visuomotor policy when deployed on a real robot. The supplementary video is available at: https://youtu.be/B6nODKkhzSw.


  • LBSNet: Lightweight Joint Boundary Detection and Semantic Segmentation for Transparent and Reflective Objects

    by Ling Tong, Kun Qian, Xingshuo Jing

    Accurate visual detection of transparent and reflective objects remains a challenging issue for mobile manipulators. For the most common depth cameras and LiDAR sensors, the distinctive optical attributes inherent in both transparent and reflective objects pose a significant challenge. To address this problem, this study proposes a lightweight joint boundary detection and semantic segmentation network named LBSNet. LBSNet aims to enhance the perception of transparent and reflective objects in complex and dynamic environments, using RGB images only. It leverages the synergy between boundary detection and semantic segmentation through feature fusion and a multitask learning mechanism. The encoder consists of two paths: one captures category-aware semantic information, while the other discerns boundaries from multi-scale features. The gated channel adaptive (GCA) module enhances boundary features by learning channel parameters. The dynamic adaptive feature fusion (DAFF) module dynamically adjusts semantic and boundary information through cross-feature fusion. These methods effectively capture the distinctive characteristics of transparent and reflective objects, such as light refraction, boundary blurring and low contrast. Experimental results show that LBSNet achieves higher accuracy and faster processing speed on multiple public datasets compared with existing methods. Moreover, its lightweight design makes it suitable for resource-constrained mobile manipulators.


  • Unsupervised Adversarial Domain Adaptation for Sim-to-Real Transfer of Tactile Images

    by Xingshuo Jing, Kun Qian, Tudor Jianu, Shan Luo

    Transferring optical tactile skills learned from simulated environments to the real world benefits many robotic tactile applications, which can reduce the cost of data collection. However, the models purely trained on simulated data are often difficult to generalize well to the unseen real world due to the domain gap between the faultless training images and testing images with unpredictable manufacturing defects or natural wear. In this paper, we propose an Adaptively Correlation-attentive and Task-related Network (ACTNet) for tactile image transfer, a novel unsupervised adversarial domain adaptation method to narrow the domain gap for pixel-level tactile perception tasks. An adaptively correlative attention mechanism is introduced to improve the generator, which is capable of leveraging global information and focusing on salient regions. We also construct a task-related constraint loss based on the tactile image classification task to facilitate the zero-shot sim-to-real transfer of the tactile image classifier. Extensive experiment results on tactile image generation and tactile image classification indicate that our method outperforms the state-of-the-art method. Classification accuracy of 92.85% in the real-world environment is achieved without using any real-world classification labels.


  • Pixel-Level Domain Adaptation for Real-to-Sim Object Pose Estimation

    by Kun Qian, Yanhui Duan, Chaomin Luo, Yongqiang Zhao, Xingshuo Jing

    Transferring robotic grasping skills learned from a simulator to the real world is beneficial in reducing the cost of labeling. However, the models trained on synthetic data are brittle when being applied to real-world data due to the domain gap. In this paper, we propose CCM Pixel-DA, a novel real-to-sim approach to unsupervised domain adaptation for object pose estimations, which outperforms conventional domain adaptation methods in preserving structural information, semantic information, and object pose during the transfer. The pipeline decouples domain adaptation and pose estimation, which allows the CCM Pixel-DA method to be integrated into state-of-the-art object pose estimation networks. The proposed method is further integrated into a pipeline for robot grasping. Experimental results on a real-world robot grasping system validate that the system is capable of grasping real-world objects without object pose annotations in the real-world domain.


  • Learning Robot Manipulation Skills From Human Demonstration Videos Using Two-Stream 2-D/3-D Residual Networks With Self-Attention

    by Xin Xu, Kun Qian, Xingshuo Jing, Wei Song

    Learning manipulation skills from observing human demonstration video is a promising aspect for intelligent robotic systems. Recent advances in Video-to-Command provide an end-to-end approach to translate a video into robot plans. However, simultaneous Video-to-Command and action segmentation remain a major challenge for bimanual manipulations with fine-grained actions. Another concern is the generalization capability of end-to-end approaches in dealing with varied task parameters as well as environmental changes between the learned skills and the one-shot task demonstration for the robot to replay. In this paper, we propose a two-stream network for robots to learn and segment manipulation sub-actions from human demonstration videos. Our framework with the self-attention mechanism can segment learned skills and generate action commands simultaneously. To arrive at refined plans in situations of under-specified or redundant human demonstrations, we utilize PDDL-based skill scripts to model the semantics of demonstrated activities and infer latent movements. Experimental results on the extended manipulation dataset indicate that our approach generates more accurate commands than the state-of-the-art methods. Real-world experiment results on a Baxter robotic arm also demonstrated the feasibility of our method in reproducing fine-grained actions from video demonstrations.