ℱsign-Net: Depth Sensor Aggregated Frame-Based Fourier Network for Sign Word Recognition
- 12 June 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Sensors Journal
- Vol. 24 (22) , 37630-37645
Abstract
Hand-tracking is a challenging problem during hand gesture recognition due to abnormal hand patterns across depth signs and errors between normal pixels and backgrounds. In this article, we propose F sign-Net: Fourier pixel-wise approach based on the Fourier Convolution Neural Network (FCNN) and Time-distributed-based Bi-directional Long-short-term Memory (BiLSTM). FCNNs have been widely researched, reaching state-of-the-art performance on spatial recognition tasks. However, it is still difficult for the Fourier model to learn the temporal patterns due to the chaotic nature of the hand motion data. F sign-Net aggregates time-based information from spatial and temporal modules to a given Fourier convolution in three stages: (1) Each depth frame is regarded as a window, and it is selected so that the aggregated sums of the pixels across the hand joints of the selected window are aligned, (2) a truncated pooling is applied that summarizes the generated featured map of the Fourier convolution to avoid over-fitting, and (3) the long-term temporal dependencies among pixels for the Fourier convolution are captured using the shared Time-based BiLSTM layers. This allows the proposed model to learn hand patterns that are temporally oriented. Finally, the proposed F net-Sign is evaluated on depth sign language public data sets and demonstrates state-of-the-art performance. Simulation results proved that improved Fourier features are good features for the proposed hand-tracking approach.Keywords
Funding Information
- King Mongkut’s University of Technology Thonburi’s Postdoctoral Fellowship
This publication has 41 references indexed in Scilit:
- Automated Indian sign language recognition system by fusing deep and handcrafted featureMultimedia Tools and Applications, 2022
- A deep sign language recognition system for Indian sign languageNeural Computing and Applications, 2022
- Research on the improved gesture tracking algorithm in sign language synthesisThe Journal of Supercomputing, 2022
- An Efficient Sign Language Recognition (SLR) System Using Camshift Tracker and Hidden Markov Model (HMM)SN Computer Science, 2021
- Arabic sign language recognition using Ada-Boosting based on a leap motion controllerInternational Journal of Information Technology, 2020
- Exploiting 3D Hand Pose Estimation in Deep Learning-Based Sign Language Recognition from RGB Videos2020
- Combining Static and Dynamic Predictions of Transfer Points for Human Initiated Handovers2019
- Deep Learning Based Hand Gesture Recognition and UAV Flight ControlsInternational Journal of Automation and Computing, 2019
- FCNN: Fourier Convolutional Neural Networks2017
- Hand gesture recognition with jointly calibrated Leap Motion and depth sensorMultimedia Tools and Applications, 2015