Canonical Representation and Force-Based Pretraining of 3D Tactile for Dexterous Visuo-Tactile Policy Learning
Authors:
- 19 May 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 6786-6792
Abstract
Tactile sensing plays a vital role in enabling robots to perform fine-grained, contact-rich tasks. However, the high dimensionality of tactile data, due to the large coverage on dexterous hands, poses significant challenges for effective tactile feature learning, especially for 3D tactile data, as there are no large standardized datasets and no strong pretrained backbones. To address these challenges, we propose a novel canonical representation that reduces the difficulty of 3D tactile feature learning and further introduces a force-based selfsupervised pretraining task to capture both local and net force features, which are crucial for dexterous manipulation. Our method achieves an average success rate of 78% across four fine-grained, contact-rich dexterous manipulation tasks in realworld experiments, demonstrating effectiveness and robustness compared to other methods. Further analysis shows that our method fully utilizes both spatial and force information from 3D tactile data to accomplish the tasks. The videos can be viewed at https://3dtacdex.github.io.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62376006)
This publication has 27 references indexed in Scilit:
- Reconstructing Hands in 3D with Transformers2024
- Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Training2024
- Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing2024
- Digger Finger: GelSight Tactile Sensor for Object Identification Inside Granular Media2021
- DexPilot: Vision-Based Teleoperation of Dexterous Robotic Hand-Arm System2020
- Connecting Touch and Vision via Cross-Modal Prediction2019
- Learning the signatures of the human grasp using a scalable tactile gloveNature, 2019
- ViTac: Feature Sharing Between Vision and Tactile Sensing for Cloth Texture Recognition2018
- Connecting Look and Feel: Associating the Visual and Tactile Properties of Physical Materials2017
- Somatosensory control of precision grip during unpredictable pulling loadsExperimental Brain Research, 1992