Abstract
To realize the exquisite interaction and precise manipulation for the robot, in this article, we propose a multispectral multimodal visuotactile sensor named M $^{3}$ Tac, which combines visible, near-infrared, and mid-infrared imaging technologies for the first time and can exceed the sensing ability of human skin in terms of resolution (719 pixels/cm $^{2}$ ), temperature sensing range (-20 $\sim$ 130 $^\text{o}$ C), etc. The M $^{3}$ Tac can not only realize high-quality sensing of deformation, texture, force, stickiness, and temperature comparable to human skin but also can realize proximity sensing that is lacking for human skin. To achieve this, we not only design a multispectral imaging system with an elastic film whose light penetrability can be regulated by the brightness of the light, but also develop corresponding algorithms, including the pixel-level force sensing with finite element method (accuracy: $\pm$ 0.023 N), the proximity perception (accuracy: $\pm$ 3.8 mm), the 3D reconstruction (accuracy: 0.33 mm), the super-resolution temperature sensing (accuracy: $\pm$ 0.3 $^\text{o}$ C), the multimodal fusion classification (accuracy: 98%), and the stickiness recognition (accuracy: 98%). Finally, we conduct experiments to verify the effectiveness and application potential of our research. This paper has supplementary material available at https://sites.google.com/view/MTac-sensor .
Funding Information
  • Shenzhen Key Laboratory of Ubiquitous Data Enabling (ZDSYS20220527171406015)
  • Shenzhen Science and Technology Program (JCYJ20220530143013030)
  • Guangdong Innovative and Entrepreneurial Research Team Program (2021ZT09L197)
  • National Natural Science Foundation of China (62104125, 62003188)
  • Shenzhen Higher Education Stable Support Program (WDZC20231129093657002)
  • Tsinghua Shenzhen International Graduate School-Shenzhen Pengrui Young Faculty Program of Shenzhen Pengrui Foundation (SZPR2023005)