Video Sign Language Recognition using Pose Extraction and Deep Learning Models
Author:
Open Access
- 15 June 2023
- thesis
- Published by San Jose State University Library
Abstract
Sign language recognition (SLR) has long been a studied subject and research field within the Computer Vision domain. Appearance-based and pose-based approaches are two ways to tackle SLR tasks. Various models from traditional to current state-of-the-art including HOG-based features, Convolutional Neural Network, Recurrent Neural Network, Transformer, and Graph Convolutional Network have been utilized to tackle the area of SLR. While classifying alphabet letters in sign language has shown high accuracy rates, recognizing words presents its set of difficulties including the large vocabulary size, the subtleties in body motions and hand orientations, and regional dialects and variations. The emergence of deep learning has created opportunities for improved word-level sign recognition, but challenges such as overfitting and limited training data remain. Techniques such as data augmentation, feature engineering, hyperparameter tuning, optimization, and ensemble methods have been used to overcome these challenges and improve the accuracy and generalization ability of ASL classification models. We explore various methods to improve the accuracy and performance in this project. From the approach, we were able to first reproduce a baseline accuracy of 43.02% on the WLASL dataset and further achieve an improvement in accuracy at 55.96%. We also extended the work to a different dataset to gain a comprehensive understanding of our work.Keywords
This publication has 10 references indexed in Scilit:
- Hierarchical Contrast for Unsupervised Skeleton-Based Action Representation LearningProceedings of the AAAI Conference on Artificial Intelligence, 2023
- Multi-cue temporal modeling for skeleton-based sign language recognitionFrontiers in Neuroscience, 2023
- OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages2022
- American Sign Language Alphabet Recognition by Extracting Feature from Hand Pose EstimationSensors, 2021
- AUTSL: A Large Scale Multi-Modal Turkish Sign Language Dataset and Baseline MethodsIEEE Access, 2020
- A Deep Neural Framework for Continuous Sign Language Recognition by Iterative TrainingIEEE Transactions on Multimedia, 2019
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization2017
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- Learning sign language by watching TV (using weakly aligned subtitles)2009