Abstract
In this article, we present a pneumatic soft hand with integrated tactile sensing, state estimation, and embodied intelligence capabilities. A novel state estimation method is proposed based on the linear elasticity theory. Following the theoretical principles above, the hardware for the soft hand is developed, which includes the fabrication of soft fingers with multiple sensors and the design of controller circuits. The soft hand achieves real-time tactile sensing for all three fingers at 7 Hz. A multimodal state estimation neural network is proposed to learn the principles of state estimation. It estimates the 3-D finger state with a 1.73 mm average error in 4.21 ms when deployed on the soft hand controller with a Jetson Orin Nano.Embodied intelligence in the soft hand is realized through vision-language models using a hierarchical pyramid prompt architecture, enabling the hand to perform in-hand manipulations using only the movements of its soft fingers, achieving an average success rate of 92.5%. The prompts, 3-D models, PCB images, source code, and video demos are provided in the supplemental materials attached to this article.
Funding Information
  • Deutsche Forschungsgemeinschaft (DFG TRR-169)
  • HiTeC e.V. at the University of Hamburg
  • National Natural Science Foundation of China (92048205)