HearASL: Your Smartphone Can Hear American Sign Language
- 27 December 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 10 (10) , 8839-8852
Abstract
Sign language is expressed by movements of the hands and facial expressions, which is mainly used by the deaf community. Although some gesture recognition methods are put forward, they possess different defects and are not applicable to deal with the sign language recognition (SLR) problem. In this article, we propose an end-to-end American SLR system with built-in speakers and microphones in smartphones, which enables SLR at both word level and sentence level. The high-level idea is to use the inaudible acoustic signal to estimate channel information and capture the sign language in real time. We use channel impulse response to represent each sign language gesture, which can realize finger-level recognition. We also pay attention to conversion movements between two words and treat them as an additional label when training the sentence-level classification model. We implement a prototype system and run a series of experiments that demonstrate the promising performance of our system. Experimental results show that our approach can achieve an accuracy of 97.2% at word-level recognition and word error rate of 0.9% at sentence-level recognition, respectively.Keywords
Funding Information
- National Natural Science Foundation of China (62072040)
- National Natural Science Foundation of China (61902212)
- Beijing Institute of Technology Research Fund Program for Young Scholars
This publication has 28 references indexed in Scilit:
- A mobile application for South African Sign Language (SASL) recognition2015
- A real-time portable sign language translation system2015
- Real-time American Sign Language Recognition using wrist-worn motion and surface EMG sensors2015
- American Sign Language Recognition Using Leap Motion Sensor2014
- Real-Time Sign Language Recognition Using a Consumer Depth Camera2013
- A Sign-Component-Based Framework for Chinese Sign Language Recognition Using Accelerometer and sEMG DataIEEE Transactions on Biomedical Engineering, 2012
- Sign Language Recognition Using Kinect2012
- American sign language recognition with the kinect2011
- Object-based visual attention for computer visionArtificial Intelligence, 2003
- Sign language recognition using sensor gloves2002