HXHuazhe Xu

Huazhe Xu

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Huazhe Xu has a total of 60 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • MoMaBond: Bridging Navigation and Manipulation Through Object-Centric Alignment

    by Widyadewi Soedarmdji, Zhengrong Xue, Yuchen Liu, Xuemei Wang, Ziming Wang, Tianhai Liang, Shiyi Guo, Huazhe Xu

  • X-Distill: Cross-Architecture Vision Distillation for Visuomotor Learning

    by Maanping Shao, Feihong Zhang, Gu Zhang, Baiye Cheng, Zhengrong Xue, Huazhe Xu

  • Top Cited Papers
    Open Access

    Chang-E: A High-Quality Motion Capture Dataset of Chinese Classical Dunhuang Dance

    by Zeyu Wang, Chengan He, Zhe Yan, Jiashun Wang, Yingke Wang, Junhua Liu, Angela Shen, Mengying Zeng, Holly Rushmeier, Huazhe Xu, et al.

    Derived from the mural drawings in the UNESCO-listed Mogao Caves, Dunhuang dance has unique cultural value but faces challenges of digitization and preservation. In this article, we introduce the first open comprehensive motion capture dataset of Dunhuang dance, Chang-E , including full-body movements documented across 8 categories, totaling 40 minutes of professional dance (preview available at https://cislab.hkust-gz.edu.cn/projects/chang-e/ ). This dataset contains three formats: skeleton data acquired from motion capture, body mesh generated from skeleton using machine learning, and multiview videos recorded on site. The dataset supports various creative applications for Dunhuang dance culture, as demonstrated by an immersive new media exhibition. Through the curation process, we applied motion inbetweening algorithms to concatenate different dance sequences for choreography. Also, these reinterpreted dance sequences are synchronized with music using retiming techniques, augmenting the rhythms and harmony between the music and dance performance. Furthermore, we applied visual effects on the regenerated motion sequences of digital dancers, achieving artistic and appealing visual results echoing Buddhist discourses of meditation and bodily cognition. The Chang-E dataset enables digital preservation and creative reimagination of Dunhuang dance, offering not only high-quality data but also an interdisciplinary collaboration framework for future graphics and cultural heritage research.


  • RoboDuet: Learning a Cooperative Policy for Whole-Body Legged Loco-Manipulation

    by Guoping Pan, Qingwei Ben, Zhecheng Yuan, Guangqi Jiang, Yandong Ji, Shoujie Li, Jiangmiao Pang, Houde Liu, Huazhe Xu

    Fully leveraging the loco-manipulation capabilities of a quadruped robot equipped with a robotic arm is non-trivial, as it requires controlling all degrees of freedom (DoFs) of the quadruped robot to achieve effective whole-body coordination. In this letter, we propose a novel framework RoboDuet, which employs two collaborative policies to realize locomotion and manipulation simultaneously, achieving whole-body control through mutual interactions. Beyond enabling large-range 6D pose tracking for manipulation, we find that the two-policy framework supports zero-shot transfer across quadruped robots with similar morphology and physical dimensions in the real world. Our experiments demonstrate that RoboDuet achieves a 23% improvement in success rate over the baseline in challenging loco-manipulation tasks employing whole-body control. To support further research, we provide open-source code and additional videos on our website: https://locomanip-duet.github.io/.


  • Top Cited Papers

    9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation

    by Changyi Lin, Han Zhang, Jikai Xu, Lei Wu, Huazhe Xu

    The advancements in vision-based tactile sensors have boosted the aptitude of robots to perform contact-rich manipulation, particularly when precise positioning and contact state of the manipulated objects are crucial for successful execution. In this work, we present 9D Tact, a straightforward yet versatile tactile sensor that offers 3D shape reconstruction and 6D force estimation capabilities. Conceptually, 9DTact is designed to be highly compact, robust, and adaptable to various robotic platforms. Moreover, it is low-cost and easy-to-fabricate, requiring minimal assembly skills. Functionally, 9DTact builds upon the optical principles of DTact and is optimized to achieve 3D shape reconstruction with enhanced accuracy and efficiency. Remarkably, we leverage the optical and deformable properties of the translucent gel so that 9DTact can perform 6D force estimation without the participation of auxiliary markers or patterns on the gel surface. More specifically, we collect a dataset consisting of approximately 100 000 image-force pairs from 175 complex objects and train a neural network to regress the 6D force, which can generalize to unseen objects.To promote the development and applications of vision-based tactile sensors, we open-source both the hardware and software of 9DTact, along with a comprehensive video tutorial.


  • Open Access

    USEEK: Unsupervised SE(3)-Equivariant 3D Keypoints for Generalizable Manipulation

    by Zhengrong Xue, Zhecheng Yuan, Jiashun Wang, Xueqian Wang, Yang Gao, Huazhe Xu

    Can a robot manipulate intra-category unseen objects in arbitrary poses with the help of a mere demonstration of grasping pose on a single object instance? In this paper, we try to address this intriguing challenge by using USEEK, an unsupervised SE(3)-equivariant keypoints method that enjoys alignment across instances in a category, to perform generalizable manipulation. USEEK follows a teacher-student structure to decouple the unsupervised keypoint discovery and SE(3)-equivariant keypoint detection. With USEEK in hand, the robot can infer the category-level task-relevant object frames in an efficient and explainable manner, enabling manipulation of any intra-category objects from and to any poses. Through extensive experiments, we demonstrate that the keypoints produced by USEEK possess rich semantics, thus successfully transferring the functional knowledge from the demonstration object to the novel ones. Compared with other object representations for manipulation, USEEK is more adaptive in the face of large intra-category shape variance, more robust with limited demonstrations, and more efficient at inference time.

  • Open Access

    Towards Learning to Play Piano with Dexterous Hands and Touch

    by Huazhe Xu, Yuping Luo, Shaoxiong Wang, Trevor Darrell, Roberto Calandra

    As Liszt once said “(a virtuoso) must call up scent and blossom, and breathe the breath of life”, a virtuoso plays the piano with passion, poetry, and extraordinary technical ability. Hence, piano playing, being a task that is quintessentially human, becomes a hallmark for roboticians and artificial intelligence researchers to pursue. In this paper, we advocate an end-to-end reinforcement learning (RL) paradigm to demonstrate how an agent can learn directly from machine-readable music score to play the piano with touch-augmented dexterous hands on a simulated piano. To achieve the desired tasks, we design useful touch- and audio-based reward functions and a series of tasks. Empirical results show that the RL agent can not only find the correct key position but also deal with the various rhythmic, volume, and fingering requirements. As a result, the agent demonstrates its effectiveness in playing simple pieces that have different musical requirements which show the potential of leveraging reinforcement learning approach for the piano playing tasks.


  • Open Access

    RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks

    by Haochen Shi, Huazhe Xu, Zhiao Huang, Yunzhu Li, Jiajun Wu

    Modeling and manipulating elasto-plastic objects are essential capabilities for robots to perform complex industrial and household interaction tasks (e.g., stuffing dumplings, rolling sushi, and making pottery). However, due to the high degree of freedom of elasto-plastic objects, significant challenges exist in virtually every aspect of the robotic manipulation pipeline, e.g., representing the states, modeling the dynamics, and synthesizing the control signals. We propose to tackle these challenges by employing a particle-based representation for elasto-plastic objects in a model-based planning framework. Our system, RoboCraft, only assumes access to raw RGBD visual observations. It transforms the sensing data into particles and learns a particle-based dynamics model using graph neural networks (GNNs) to capture the structure of the underlying system. The learned model can then be coupled with model-predictive control (MPC) algorithms to plan the robot's behavior. We show through experiments that with just 10 minutes of real-world robotic interaction data, our robot can learn a dynamics model that can be used to synthesize control signals to deform elasto-plastic objects into various target shapes, including shapes that the robot has never encountered before. We perform systematic evaluations in both simulation and the real world to demonstrate the robot's manipulation capabilities and ability to generalize to a more complex action space, different tool shapes, and a mixture of motion modes. We also conduct comparisons between RoboCraft and untrained human subjects controlling the gripper to manipulate deformable objects in both simulation and the real world. Our learned model-based planning framework is comparable to and sometimes better than human subjects on the tested tasks.