Huazhe Xu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Huazhe Xu has a total of 60 co-authors. Here are the top co-authors by number of shared publications:
Zhecheng Yuan
Tsinghua University, Beijing, China
3 shared publications- Zhengrong Xue
Tsinghua University
3 shared publications Tianhai Liang
Tsinghua University
2 shared publications- Xueqian Wang
Tsinghua University
2 shared publications - Angela Shen
Northeastern University, Boston, MA, USA
1 shared publication Baiye Cheng
Huazhong University of Science and Technology
1 shared publication
Latest Publications
- 1 January 2027
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 12 (1) , 73-80
MoMaBond: Bridging Navigation and Manipulation Through Object-Centric Alignment
- 1 January 2027
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 12 (1) , 73-80
MoMaBond: Bridging Navigation and Manipulation Through Object-Centric Alignment
- 1 September 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 11 (9) , 10672-10679
X-Distill: Cross-Architecture Vision Distillation for Visuomotor Learning
- 1 September 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 11 (9) , 10672-10679
X-Distill: Cross-Architecture Vision Distillation for Visuomotor Learning
- Top Cited PapersOpen Access
- 22 January 2026
- Published by Association for Computing Machinery (ACM) in Journal on Computing and Cultural Heritage
- Vol. 19 (1) , 1-20
Chang-E: A High-Quality Motion Capture Dataset of Chinese Classical Dunhuang Dance
Derived from the mural drawings in the UNESCO-listed Mogao Caves, Dunhuang dance has unique cultural value but faces challenges of digitization and preservation. In this article, we introduce the first open comprehensive motion capture dataset of Dunhuang dance, Chang-E , including full-body movements documented across 8 categories, totaling 40 minutes of professional dance (preview available at https://cislab.hkust-gz.edu.cn/projects/chang-e/ ). This dataset contains three formats: skeleton data acquired from motion capture, body mesh generated from skeleton using machine learning, and multiview videos recorded on site. The dataset supports various creative applications for Dunhuang dance culture, as demonstrated by an immersive new media exhibition. Through the curation process, we applied motion inbetweening algorithms to concatenate different dance sequences for choreography. Also, these reinterpreted dance sequences are synchronized with music using retiming techniques, augmenting the rhythms and harmony between the music and dance performance. Furthermore, we applied visual effects on the regenerated motion sequences of digital dancers, achieving artistic and appealing visual results echoing Buddhist discourses of meditation and bodily cognition. The Chang-E dataset enables digital preservation and creative reimagination of Dunhuang dance, offering not only high-quality data but also an interdisciplinary collaboration framework for future graphics and cultural heritage research.
- 22 January 2026
- Published by Association for Computing Machinery (ACM) in Journal on Computing and Cultural Heritage
- Vol. 19 (1) , 1-20
Top Cited PapersOpen AccessChang-E: A High-Quality Motion Capture Dataset of Chinese Classical Dunhuang Dance
Derived from the mural drawings in the UNESCO-listed Mogao Caves, Dunhuang dance has unique cultural value but faces challenges of digitization and preservation. In this article, we introduce the first open comprehensive motion capture dataset of Dunhuang dance, Chang-E , including full-body movements documented across 8 categories, totaling 40 minutes of professional dance (preview available at https://cislab.hkust-gz.edu.cn/projects/chang-e/ ). This dataset contains three formats: skeleton data acquired from motion capture, body mesh generated from skeleton using machine learning, and multiview videos recorded on site. The dataset supports various creative applications for Dunhuang dance culture, as demonstrated by an immersive new media exhibition. Through the curation process, we applied motion inbetweening algorithms to concatenate different dance sequences for choreography. Also, these reinterpreted dance sequences are synchronized with music using retiming techniques, augmenting the rhythms and harmony between the music and dance performance. Furthermore, we applied visual effects on the regenerated motion sequences of digital dancers, achieving artistic and appealing visual results echoing Buddhist discourses of meditation and bodily cognition. The Chang-E dataset enables digital preservation and creative reimagination of Dunhuang dance, offering not only high-quality data but also an interdisciplinary collaboration framework for future graphics and cultural heritage research.
- 1 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Robotics
- Vol. 42, 3489-3512
HERMES: Human-to-Robot Embodied Learning From Multisource Motion Data for Mobile Dexterous Manipulation
- 1 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Robotics
- Vol. 42, 3489-3512
HERMES: Human-to-Robot Embodied Learning From Multisource Motion Data for Mobile Dexterous Manipulation
- 14 March 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 10 (5) , 4564-4571
RoboDuet: Learning a Cooperative Policy for Whole-Body Legged Loco-Manipulation
Fully leveraging the loco-manipulation capabilities of a quadruped robot equipped with a robotic arm is non-trivial, as it requires controlling all degrees of freedom (DoFs) of the quadruped robot to achieve effective whole-body coordination. In this letter, we propose a novel framework RoboDuet, which employs two collaborative policies to realize locomotion and manipulation simultaneously, achieving whole-body control through mutual interactions. Beyond enabling large-range 6D pose tracking for manipulation, we find that the two-policy framework supports zero-shot transfer across quadruped robots with similar morphology and physical dimensions in the real world. Our experiments demonstrate that RoboDuet achieves a 23% improvement in success rate over the baseline in challenging loco-manipulation tasks employing whole-body control. To support further research, we provide open-source code and additional videos on our website: https://locomanip-duet.github.io/.
- 14 March 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 10 (5) , 4564-4571
RoboDuet: Learning a Cooperative Policy for Whole-Body Legged Loco-Manipulation
Fully leveraging the loco-manipulation capabilities of a quadruped robot equipped with a robotic arm is non-trivial, as it requires controlling all degrees of freedom (DoFs) of the quadruped robot to achieve effective whole-body coordination. In this letter, we propose a novel framework RoboDuet, which employs two collaborative policies to realize locomotion and manipulation simultaneously, achieving whole-body control through mutual interactions. Beyond enabling large-range 6D pose tracking for manipulation, we find that the two-policy framework supports zero-shot transfer across quadruped robots with similar morphology and physical dimensions in the real world. Our experiments demonstrate that RoboDuet achieves a 23% improvement in success rate over the baseline in challenging loco-manipulation tasks employing whole-body control. To support further research, we provide open-source code and additional videos on our website: https://locomanip-duet.github.io/.
- Top Cited Papers
- 5 December 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9 (2) , 923-930
9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation
The advancements in vision-based tactile sensors have boosted the aptitude of robots to perform contact-rich manipulation, particularly when precise positioning and contact state of the manipulated objects are crucial for successful execution. In this work, we present 9D Tact, a straightforward yet versatile tactile sensor that offers 3D shape reconstruction and 6D force estimation capabilities. Conceptually, 9DTact is designed to be highly compact, robust, and adaptable to various robotic platforms. Moreover, it is low-cost and easy-to-fabricate, requiring minimal assembly skills. Functionally, 9DTact builds upon the optical principles of DTact and is optimized to achieve 3D shape reconstruction with enhanced accuracy and efficiency. Remarkably, we leverage the optical and deformable properties of the translucent gel so that 9DTact can perform 6D force estimation without the participation of auxiliary markers or patterns on the gel surface. More specifically, we collect a dataset consisting of approximately 100 000 image-force pairs from 175 complex objects and train a neural network to regress the 6D force, which can generalize to unseen objects.To promote the development and applications of vision-based tactile sensors, we open-source both the hardware and software of 9DTact, along with a comprehensive video tutorial.
- 5 December 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9 (2) , 923-930
Top Cited Papers9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation
The advancements in vision-based tactile sensors have boosted the aptitude of robots to perform contact-rich manipulation, particularly when precise positioning and contact state of the manipulated objects are crucial for successful execution. In this work, we present 9D Tact, a straightforward yet versatile tactile sensor that offers 3D shape reconstruction and 6D force estimation capabilities. Conceptually, 9DTact is designed to be highly compact, robust, and adaptable to various robotic platforms. Moreover, it is low-cost and easy-to-fabricate, requiring minimal assembly skills. Functionally, 9DTact builds upon the optical principles of DTact and is optimized to achieve 3D shape reconstruction with enhanced accuracy and efficiency. Remarkably, we leverage the optical and deformable properties of the translucent gel so that 9DTact can perform 6D force estimation without the participation of auxiliary markers or patterns on the gel surface. More specifically, we collect a dataset consisting of approximately 100 000 image-force pairs from 175 complex objects and train a neural network to regress the 6D force, which can generalize to unseen objects.To promote the development and applications of vision-based tactile sensors, we open-source both the hardware and software of 9DTact, along with a comprehensive video tutorial.
- Open Access
USEEK: Unsupervised SE(3)-Equivariant 3D Keypoints for Generalizable Manipulation
Can a robot manipulate intra-category unseen objects in arbitrary poses with the help of a mere demonstration of grasping pose on a single object instance? In this paper, we try to address this intriguing challenge by using USEEK, an unsupervised SE(3)-equivariant keypoints method that enjoys alignment across instances in a category, to perform generalizable manipulation. USEEK follows a teacher-student structure to decouple the unsupervised keypoint discovery and SE(3)-equivariant keypoint detection. With USEEK in hand, the robot can infer the category-level task-relevant object frames in an efficient and explainable manner, enabling manipulation of any intra-category objects from and to any poses. Through extensive experiments, we demonstrate that the keypoints produced by USEEK possess rich semantics, thus successfully transferring the functional knowledge from the demonstration object to the novel ones. Compared with other object representations for manipulation, USEEK is more adaptive in the face of large intra-category shape variance, more robust with limited demonstrations, and more efficient at inference time.
Open AccessUSEEK: Unsupervised SE(3)-Equivariant 3D Keypoints for Generalizable Manipulation
Can a robot manipulate intra-category unseen objects in arbitrary poses with the help of a mere demonstration of grasping pose on a single object instance? In this paper, we try to address this intriguing challenge by using USEEK, an unsupervised SE(3)-equivariant keypoints method that enjoys alignment across instances in a category, to perform generalizable manipulation. USEEK follows a teacher-student structure to decouple the unsupervised keypoint discovery and SE(3)-equivariant keypoint detection. With USEEK in hand, the robot can infer the category-level task-relevant object frames in an efficient and explainable manner, enabling manipulation of any intra-category objects from and to any poses. Through extensive experiments, we demonstrate that the keypoints produced by USEEK possess rich semantics, thus successfully transferring the functional knowledge from the demonstration object to the novel ones. Compared with other object representations for manipulation, USEEK is more adaptive in the face of large intra-category shape variance, more robust with limited demonstrations, and more efficient at inference time.
- Open Access
- 23 October 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 10410-10416
Towards Learning to Play Piano with Dexterous Hands and Touch
As Liszt once said “(a virtuoso) must call up scent and blossom, and breathe the breath of life”, a virtuoso plays the piano with passion, poetry, and extraordinary technical ability. Hence, piano playing, being a task that is quintessentially human, becomes a hallmark for roboticians and artificial intelligence researchers to pursue. In this paper, we advocate an end-to-end reinforcement learning (RL) paradigm to demonstrate how an agent can learn directly from machine-readable music score to play the piano with touch-augmented dexterous hands on a simulated piano. To achieve the desired tasks, we design useful touch- and audio-based reward functions and a series of tasks. Empirical results show that the RL agent can not only find the correct key position but also deal with the various rhythmic, volume, and fingering requirements. As a result, the agent demonstrates its effectiveness in playing simple pieces that have different musical requirements which show the potential of leveraging reinforcement learning approach for the piano playing tasks.
- 23 October 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 10410-10416
Open AccessTowards Learning to Play Piano with Dexterous Hands and Touch
As Liszt once said “(a virtuoso) must call up scent and blossom, and breathe the breath of life”, a virtuoso plays the piano with passion, poetry, and extraordinary technical ability. Hence, piano playing, being a task that is quintessentially human, becomes a hallmark for roboticians and artificial intelligence researchers to pursue. In this paper, we advocate an end-to-end reinforcement learning (RL) paradigm to demonstrate how an agent can learn directly from machine-readable music score to play the piano with touch-augmented dexterous hands on a simulated piano. To achieve the desired tasks, we design useful touch- and audio-based reward functions and a series of tasks. Empirical results show that the RL agent can not only find the correct key position but also deal with the various rhythmic, volume, and fingering requirements. As a result, the agent demonstrates its effectiveness in playing simple pieces that have different musical requirements which show the potential of leveraging reinforcement learning approach for the piano playing tasks.
- Open Access
RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks
Modeling and manipulating elasto-plastic objects are essential capabilities for robots to perform complex industrial and household interaction tasks (e.g., stuffing dumplings, rolling sushi, and making pottery). However, due to the high degree of freedom of elasto-plastic objects, significant challenges exist in virtually every aspect of the robotic manipulation pipeline, e.g., representing the states, modeling the dynamics, and synthesizing the control signals. We propose to tackle these challenges by employing a particle-based representation for elasto-plastic objects in a model-based planning framework. Our system, RoboCraft, only assumes access to raw RGBD visual observations. It transforms the sensing data into particles and learns a particle-based dynamics model using graph neural networks (GNNs) to capture the structure of the underlying system. The learned model can then be coupled with model-predictive control (MPC) algorithms to plan the robot's behavior. We show through experiments that with just 10 minutes of real-world robotic interaction data, our robot can learn a dynamics model that can be used to synthesize control signals to deform elasto-plastic objects into various target shapes, including shapes that the robot has never encountered before. We perform systematic evaluations in both simulation and the real world to demonstrate the robot's manipulation capabilities and ability to generalize to a more complex action space, different tool shapes, and a mixture of motion modes. We also conduct comparisons between RoboCraft and untrained human subjects controlling the gripper to manipulate deformable objects in both simulation and the real world. Our learned model-based planning framework is comparable to and sometimes better than human subjects on the tested tasks.
Open AccessRoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks
Modeling and manipulating elasto-plastic objects are essential capabilities for robots to perform complex industrial and household interaction tasks (e.g., stuffing dumplings, rolling sushi, and making pottery). However, due to the high degree of freedom of elasto-plastic objects, significant challenges exist in virtually every aspect of the robotic manipulation pipeline, e.g., representing the states, modeling the dynamics, and synthesizing the control signals. We propose to tackle these challenges by employing a particle-based representation for elasto-plastic objects in a model-based planning framework. Our system, RoboCraft, only assumes access to raw RGBD visual observations. It transforms the sensing data into particles and learns a particle-based dynamics model using graph neural networks (GNNs) to capture the structure of the underlying system. The learned model can then be coupled with model-predictive control (MPC) algorithms to plan the robot's behavior. We show through experiments that with just 10 minutes of real-world robotic interaction data, our robot can learn a dynamics model that can be used to synthesize control signals to deform elasto-plastic objects into various target shapes, including shapes that the robot has never encountered before. We perform systematic evaluations in both simulation and the real world to demonstrate the robot's manipulation capabilities and ability to generalize to a more complex action space, different tool shapes, and a mixture of motion modes. We also conduct comparisons between RoboCraft and untrained human subjects controlling the gripper to manipulate deformable objects in both simulation and the real world. Our learned model-based planning framework is comparable to and sometimes better than human subjects on the tested tasks.