Can Zhao
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Can Zhao has a total of 18 co-authors. Here are the top co-authors by number of shared publications:
- Daolin Ma
School of Ocean and Civil Engineering, Shanghai Jiao Tong University, Shanghai, China
5 shared publications - Jin Liu
School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China
3 shared publications - Weiming Wang
School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China
2 shared publications - Aiguo Song
Southeast University
1 shared publication - Bidan Huang
Tencent Robotics X, Shenzhen, China
1 shared publication - Ge Gao
Mech-Mind Robotics Ltd., Beijing, China
1 shared publication
Latest Publications
- 1 July 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 11 (7) , 8752-8759
HIVE-6D: Hierarchical Visual-Tactile In-Hand Pose Estimation and Tracking
- 1 July 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 11 (7) , 8752-8759
HIVE-6D: Hierarchical Visual-Tactile In-Hand Pose Estimation and Tracking
- 8 December 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ASME Transactions on Mechatronics
- Vol. 31 (3) , 2817-2828
TACSEM: Tactile Sensing, 3-D State Estimation and In-Hand Manipulation of a Soft Hand
In this article, we present a pneumatic soft hand with integrated tactile sensing, state estimation, and embodied intelligence capabilities. A novel state estimation method is proposed based on the linear elasticity theory. Following the theoretical principles above, the hardware for the soft hand is developed, which includes the fabrication of soft fingers with multiple sensors and the design of controller circuits. The soft hand achieves real-time tactile sensing for all three fingers at 7 Hz. A multimodal state estimation neural network is proposed to learn the principles of state estimation. It estimates the 3-D finger state with a 1.73 mm average error in 4.21 ms when deployed on the soft hand controller with a Jetson Orin Nano.Embodied intelligence in the soft hand is realized through vision-language models using a hierarchical pyramid prompt architecture, enabling the hand to perform in-hand manipulations using only the movements of its soft fingers, achieving an average success rate of 92.5%. The prompts, 3-D models, PCB images, source code, and video demos are provided in the supplemental materials attached to this article.
- 8 December 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ASME Transactions on Mechatronics
- Vol. 31 (3) , 2817-2828
TACSEM: Tactile Sensing, 3-D State Estimation and In-Hand Manipulation of a Soft Hand
In this article, we present a pneumatic soft hand with integrated tactile sensing, state estimation, and embodied intelligence capabilities. A novel state estimation method is proposed based on the linear elasticity theory. Following the theoretical principles above, the hardware for the soft hand is developed, which includes the fabrication of soft fingers with multiple sensors and the design of controller circuits. The soft hand achieves real-time tactile sensing for all three fingers at 7 Hz. A multimodal state estimation neural network is proposed to learn the principles of state estimation. It estimates the 3-D finger state with a 1.73 mm average error in 4.21 ms when deployed on the soft hand controller with a Jetson Orin Nano.Embodied intelligence in the soft hand is realized through vision-language models using a hierarchical pyramid prompt architecture, enabling the hand to perform in-hand manipulations using only the movements of its soft fingers, achieving an average success rate of 92.5%. The prompts, 3-D models, PCB images, source code, and video demos are provided in the supplemental materials attached to this article.
- 14 July 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 10 (9) , 8914-8921
Tactile-Driven Dexterous In-Hand Writing via Extrinsic Contact Sensing
Dexterous in-hand manipulation, especially involving interactions between grasped objects and external environments, remains a formidable challenge in robotics. This study tackles the complexities of in-hand manipulation under extrinsic contact through a representative three-finger handwriting task. We propose a hybrid arm-hand coordination framework that combines reinforcement learning with compliance control, offering both flexibility and robustness. Leveraging tactile sensors embedded in each finger, our tactile-driven estimation model dynamically predicts in-hand object pose and external contact, eliminating the need for fixed contact states. The proposed framework is first validated in simulation, where it successfully executes diverse writing tasks with accurate contact sensing. Sim-to-Real transfer is achieved through systematic calibration of finger joints and tactile sensors, supported by domain randomization. Real-world experiments further demonstrate the system's adaptability to writing tools with varying physical properties—such as radius, length, mass, and friction—while maintaining stability across different trajectories. Also see https://inhandwriting.github.io/. This work advances robotic manipulation capabilities in unstructured environments.
- 14 July 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 10 (9) , 8914-8921
Tactile-Driven Dexterous In-Hand Writing via Extrinsic Contact Sensing
Dexterous in-hand manipulation, especially involving interactions between grasped objects and external environments, remains a formidable challenge in robotics. This study tackles the complexities of in-hand manipulation under extrinsic contact through a representative three-finger handwriting task. We propose a hybrid arm-hand coordination framework that combines reinforcement learning with compliance control, offering both flexibility and robustness. Leveraging tactile sensors embedded in each finger, our tactile-driven estimation model dynamically predicts in-hand object pose and external contact, eliminating the need for fixed contact states. The proposed framework is first validated in simulation, where it successfully executes diverse writing tasks with accurate contact sensing. Sim-to-Real transfer is achieved through systematic calibration of finger joints and tactile sensors, supported by domain randomization. Real-world experiments further demonstrate the system's adaptability to writing tools with varying physical properties—such as radius, length, mass, and friction—while maintaining stability across different trajectories. Also see https://inhandwriting.github.io/. This work advances robotic manipulation capabilities in unstructured environments.
- 19 November 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Robotics
- Vol. 41 (15523098) , 289-305
iFEM2.0: Dense 3-D Contact Force Field Reconstruction and Assessment for Vision-Based Tactile Sensors
Vision-based tactile sensors offer rich tactile information through high-resolution tactile images, enabling the reconstruction of dense contact force fields on the sensor surface. However, accurately reconstructing the 3-D contact force distribution remains a challenge. In this article, we propose the multilayer inverse finite-element method (iFEM2.0) as a robust and generalized approach to reconstruct dense contact force distribution. We systematically analyze various parameters within the iFEM2.0 framework, and determine the appropriate parameter combinations through simulation and in situ mechanical calibration. Our approach incorporates multilayer mesh constraints and ridge regularization to enhance robustness. Furthermore, as no off-the-shelf measurement equipment or criterion metrics exist for 3-D contact force distribution perception, we present a benchmark covering accuracy, fidelity, and noise resistance that can serve as a cornerstone for other future force distribution reconstruction methods. The proposed iFEM2.0 demonstrates good performance in both simulation- and experiment-based evaluations. Such dense 3-D contact force information is critical for enabling dexterous robotic manipulation that handles both rigid and soft materials.
- 19 November 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Robotics
- Vol. 41 (15523098) , 289-305
iFEM2.0: Dense 3-D Contact Force Field Reconstruction and Assessment for Vision-Based Tactile Sensors
Vision-based tactile sensors offer rich tactile information through high-resolution tactile images, enabling the reconstruction of dense contact force fields on the sensor surface. However, accurately reconstructing the 3-D contact force distribution remains a challenge. In this article, we propose the multilayer inverse finite-element method (iFEM2.0) as a robust and generalized approach to reconstruct dense contact force distribution. We systematically analyze various parameters within the iFEM2.0 framework, and determine the appropriate parameter combinations through simulation and in situ mechanical calibration. Our approach incorporates multilayer mesh constraints and ridge regularization to enhance robustness. Furthermore, as no off-the-shelf measurement equipment or criterion metrics exist for 3-D contact force distribution perception, we present a benchmark covering accuracy, fidelity, and noise resistance that can serve as a cornerstone for other future force distribution reconstruction methods. The proposed iFEM2.0 demonstrates good performance in both simulation- and experiment-based evaluations. Such dense 3-D contact force information is critical for enabling dexterous robotic manipulation that handles both rigid and soft materials.
- 8 May 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 73 (00189456) , 1-13
Real-Time Reconstruction of 3-D Tactile Motion Field via Multitask Learning
The 3-D motion field on the surface of the vision-based tactile sensor contains rich tactile information and serves as the foundation for many downstream tasks. However, achieving real-time and precise reconstruction of 3-D motion field is challenging. In this study, the 3-D motion field is decomposed into depth and 2-D motion field, and a multitask learning model is employed to simultaneously estimate both. The approach excels in accuracy and real-time performance (9.20 ms/frame). For depth reconstruction, the model achieves an average mean absolute error (MAE) of 0.062 mm. For 2-D motion field tracking, an effective structured marker tracking algorithm (SMTA) is introduced, and a marker tracking network (MaTnet) is constructed based on it. This network features dynamic sensing fields and a distance field (DF) auxiliary task, offering strong generalization, interpretability, and ease of training. It exhibits excellent tracking performance in various deformations, with an average error of 1.044 pixels (about 0.03 mm). Finally, the strong transferability of the multitask model is demonstrated, with a maximum decrease in depth reconstruction accuracy of 0.018 mm.
- 8 May 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 73 (00189456) , 1-13
Real-Time Reconstruction of 3-D Tactile Motion Field via Multitask Learning
The 3-D motion field on the surface of the vision-based tactile sensor contains rich tactile information and serves as the foundation for many downstream tasks. However, achieving real-time and precise reconstruction of 3-D motion field is challenging. In this study, the 3-D motion field is decomposed into depth and 2-D motion field, and a multitask learning model is employed to simultaneously estimate both. The approach excels in accuracy and real-time performance (9.20 ms/frame). For depth reconstruction, the model achieves an average mean absolute error (MAE) of 0.062 mm. For 2-D motion field tracking, an effective structured marker tracking algorithm (SMTA) is introduced, and a marker tracking network (MaTnet) is constructed based on it. This network features dynamic sensing fields and a distance field (DF) auxiliary task, offering strong generalization, interpretability, and ease of training. It exhibits excellent tracking performance in various deformations, with an average error of 1.044 pixels (about 0.03 mm). Finally, the strong transferability of the multitask model is demonstrated, with a maximum decrease in depth reconstruction accuracy of 0.018 mm.