Wi-Phrase: Deep Residual-Multihead Model for WiFi Sign Language Phrase Recognition
- 1 April 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 9 (18) , 18015-18027
Abstract
Sign language (SL) is used by hearing impaired and deaf people. The WiFi-based sign language recognition (SLR) technology has attracted much attention due to its contactless nature and wide applications. Most previous SLR works are designed to recognize a single isolated sign word in data samples. However, such assumption is not realistic in practical applications, such as, in daily communication, deaf people usually express their mind through a phrase (a.k.a. a group of words) rather than an isolated word. Compared with the previous works, the sign words in a phrase have variety of length, sequential patterns, and combinations in realistic communications between deaf people. Therefore, it is challenging in exploiting the WiFi signals to accurately capture the unique patterns of SL in the previous natural setting. In this article, we propose Wi-Phrase, a multigesture context-awareness SLR system. Wi-Phrase exploits WiFi signals to translate SL to the English phrase. To achieve this, Wi-Phrase employs principal component analysis (PCA) projection to filter out the noise and convert cleaned WiFi signals to spectrogram. Then, we propose a novel Residual-MultiHead model that exploit residual learn structure to obtain local patterns of phrases and adopt multihead block to capture the global context information of phrase. To prove the advanced nature of our model, we design a WiFi-based SL phrase data set of 40 categories for experiments. Our comprehensive evaluation shows that Wi-Phrase achieves an accurate phrase recognition accuracy of 95.03%. In future, we envision Wi-Phrase could be widely used as the phrase command control system for deaf people in IoT devices.Keywords
Funding Information
- National Natural Science Foundation of China (61972263, 62073225)
- Natural Science Foundation of Guangdong Province (2019A1515011608)
This publication has 30 references indexed in Scilit:
- A Robust Sign Language Recognition System with Multiple Wi-Fi Devices2017
- Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization2017
- A position and rotation invariant framework for sign language recognition (SLR) using KinectMultimedia Tools and Applications, 2017
- On the Impact of Data Set Size in Transfer Learning Using Deep Neural Networks2016
- WiFinger2016
- Deep Residual Learning for Image Recognition2016
- Sign Language in Action2016
- Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signersComputer Vision and Image Understanding, 2015
- Leveraging directional antenna capabilities for fine-grained gesture recognition2014
- From RSSI to CSIACM Computing Surveys, 2013