AviPer: assisting visually impaired people to perceive the world with visual-tactile multimodal attention network
- 30 June 2022
- journal article
- research article
- Published by Springer Nature in CCF Transactions on Pervasive Computing and Interaction
- Vol. 4 (3) , 219-239
Abstract
Unlike able-bodied persons, it is difficult for visually impaired people, especially those in the educational age, to build a full perception of the world due to the lack of normal vision. The rapid development of AI and sensing technologies has provided new solutions to visually impaired assistance. However, to our knowledge, most previous studies focused on obstacle avoidance and environmental perception but paid less attention to educational assistance for visually impaired people. In this paper, we propose AviPer, a system that aims to assist visually impaired people to perceive the world via creating a continuous, immersive, and educational assisting pattern. Equipped with a self-developed flexible tactile glove and a webcam, AviPer can simultaneously predict the grasping object and provide voice feedback using the vision-tactile fusion classification model, when a visually impaired people is perceiving the object with his gloved hand. To achieve accurate multimodal classification, we creatively embed three attention mechanisms, namely temporal, channel-wise, and spatial attention in the model. Experimental results show that AviPer can achieve an accuracy of 99.75% in classification of 10 daily objects. We evaluated the system in a variety of extreme cases, which verified its robustness and demonstrated the necessity of visual and tactile modal fusion. We also conducted tests in the actual use scene and proved the usability and user-friendliness of the system. We opensourced the code and self-collected datasets in the hope of promoting research development and bringing changes to the lives of visually impaired people.Keywords
Funding Information
- Beijing Nova Program (Z191100001119129)
- National Natural Science Foundation of China (61702520)
This publication has 57 references indexed in Scilit:
- Multimodal Wearable Sensing for Fine-Grained Activity Recognition in HealthcareIEEE Internet Computing, 2015
- Measurement of shear and slip with a GelSight tactile sensor2015
- Localizing the object contact through matching tactile features with visual map2015
- Dynamic behavior of suture-anastomosed arteries and implications to vascular surgery operationsBioMedical Engineering OnLine, 2015
- Large-Scale Video Classification with Convolutional Neural Networks2014
- Enhancing visual perception of shape through tactile glances2013
- Blind Navigation Assistance for Visually Impaired based on Local Depth Hypothesis from a Single ImageProcedia Engineering, 2013
- Accessibility assistance for visually-impaired people in digital textsBritish Journal of Visual Impairment, 2005
- Multimodal Video Indexing: A Review of the State-of-the-artMultimedia Tools and Applications, 2005
- Humans integrate visual and haptic information in a statistically optimal fashionNature, 2002