TACSEM: Tactile Sensing, 3-D State Estimation and In-Hand Manipulation of a Soft Hand
- 8 December 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ASME Transactions on Mechatronics
- Vol. 31 (3) , 2817-2828
Abstract
In this article, we present a pneumatic soft hand with integrated tactile sensing, state estimation, and embodied intelligence capabilities. A novel state estimation method is proposed based on the linear elasticity theory. Following the theoretical principles above, the hardware for the soft hand is developed, which includes the fabrication of soft fingers with multiple sensors and the design of controller circuits. The soft hand achieves real-time tactile sensing for all three fingers at 7 Hz. A multimodal state estimation neural network is proposed to learn the principles of state estimation. It estimates the 3-D finger state with a 1.73 mm average error in 4.21 ms when deployed on the soft hand controller with a Jetson Orin Nano.Embodied intelligence in the soft hand is realized through vision-language models using a hierarchical pyramid prompt architecture, enabling the hand to perform in-hand manipulations using only the movements of its soft fingers, achieving an average success rate of 92.5%. The prompts, 3-D models, PCB images, source code, and video demos are provided in the supplemental materials attached to this article.Keywords
Funding Information
- Deutsche Forschungsgemeinschaft (DFG TRR-169)
- HiTeC e.V. at the University of Hamburg
- National Natural Science Foundation of China (92048205)
This publication has 26 references indexed in Scilit:
- Acoustic Soft Tactile Skin (AST Skin)2024
- PaLmTac: A Vision-Based Tactile Sensor Leveraging Distributed-Modality Design and Modal-Matching Recognition for Soft Hand PerceptionIEEE Journal of Selected Topics in Signal Processing, 2024
- Human-Inspired Tactile Perception System for Real-Time and Multimodal Detection of Tactile StimuliSoft Robotics, 2024
- Acoustic Soft Tactile Skin: Towards Continuous Tactile Sensing2023
- GelSight Svelte: A Human Finger-Shaped Single-Camera Tactile Robot Finger with Large Sensing Coverage and Proprioceptive Sensing2023
- A concise guide to modelling the physics of embodied intelligence in soft roboticsNature Reviews Physics, 2022
- An Active Palm Enhances Dexterity of Soft Robotic In-Hand Manipulation2021
- A Dexterous Soft Robotic Hand for Delicate In-Hand ManipulationIEEE Robotics and Automation Letters, 2020
- A soft gripper with programmable effective length, tactile and curvature sensory feedbackSmart Materials and Structures, 2020
- Bodily Aware Soft Robots: Integration of Proprioceptive and Exteroceptive Sensors2018