M$^{3}$Tac: A Multispectral Multimodal Visuotactile Sensor With Beyond-Human Sensory Capabilities
- 17 September 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Robotics
- Vol. 40 (15523098) , 4484-4503
Abstract
To realize the exquisite interaction and precise manipulation for the robot, in this article, we propose a multispectral multimodal visuotactile sensor named M $^{3}$ Tac, which combines visible, near-infrared, and mid-infrared imaging technologies for the first time and can exceed the sensing ability of human skin in terms of resolution (719 pixels/cm $^{2}$ ), temperature sensing range (-20 $\sim$ 130 $^\text{o}$ C), etc. The M $^{3}$ Tac can not only realize high-quality sensing of deformation, texture, force, stickiness, and temperature comparable to human skin but also can realize proximity sensing that is lacking for human skin. To achieve this, we not only design a multispectral imaging system with an elastic film whose light penetrability can be regulated by the brightness of the light, but also develop corresponding algorithms, including the pixel-level force sensing with finite element method (accuracy: $\pm$ 0.023 N), the proximity perception (accuracy: $\pm$ 3.8 mm), the 3D reconstruction (accuracy: 0.33 mm), the super-resolution temperature sensing (accuracy: $\pm$ 0.3 $^\text{o}$ C), the multimodal fusion classification (accuracy: 98%), and the stickiness recognition (accuracy: 98%). Finally, we conduct experiments to verify the effectiveness and application potential of our research. This paper has supplementary material available at https://sites.google.com/view/MTac-sensor .Keywords
Funding Information
- Shenzhen Key Laboratory of Ubiquitous Data Enabling (ZDSYS20220527171406015)
- Shenzhen Science and Technology Program (JCYJ20220530143013030)
- Guangdong Innovative and Entrepreneurial Research Team Program (2021ZT09L197)
- National Natural Science Foundation of China (62104125, 62003188)
- Shenzhen Higher Education Stable Support Program (WDZC20231129093657002)
- Tsinghua Shenzhen International Graduate School-Shenzhen Pengrui Young Faculty Program of Shenzhen Pengrui Foundation (SZPR2023005)
This publication has 74 references indexed in Scilit:
- Swin-Unet: Unet-Like Pure Transformer for Medical Image Segmentation2023
- Image super-resolution: A comprehensive review, recent trends, challenges and applicationsInformation Fusion, 2022
- Flexible Fringe Effect Capacitive Sensors with Simultaneous High‐Performance Contact and Non‐Contact Sensing CapabilitiesSmall Structures, 2020
- Tactile electronic skin to simultaneously detect and distinguish between temperature and pressure based on a triboelectric nanogeneratorNano Energy, 2020
- Triboelectric Nanogenerator: A Foundation of the Energy for the New EraAdvanced Energy Materials, 2018
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Light, Electromagnetic Spectrum2015
- Recent Progress in Electronic SkinAdvanced Science, 2015
- Finite element formulation of the Ogden material model with application to rubber-like shellsInternational Journal for Numerical Methods in Engineering, 1998
- The bulk modulus of rubberPolymer, 1994