Siformer: Feature-isolated Transformer for Efficient Skeleton-based Sign Language Recognition
Open Access
- 28 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 9387-9396
Abstract
Sign language recognition (SLR) refers to interpreting sign language glosses from given videos automatically. This research area presents a complex challenge in computer vision because of the rapid and intricate movements inherent in sign languages, which encompass hand gestures, body postures, and even facial expressions. Recently, skeleton-based action recognition has attracted increasing attention due to its ability to handle variations in subjects and backgrounds independently. However, current skeleton-based SLR methods exhibit three limitations: 1) they often neglect the importance of realistic hand poses, where most studies train SLR models on non-realistic skeletal representations; 2) they tend to assume complete data availability in both training or inference phases, and capture intricate relationships among different body parts collectively; 3) these methods treat all sign glosses uniformly, failing to account for differences in complexity levels regarding skeletal representations. To enhance the realism of hand skeletal representations, we present a kinematic hand pose rectification method for enforcing constraints. Mitigating the impact of missing data, we propose a feature-isolated mechanism to focus on capturing local spatial-temporal context. This method captures the context concurrently and independently from individual features, thus enhancing the robustness of the SLR model. Additionally, to adapt to varying complexity levels of sign glosses, we develop an input-adaptive inference approach to optimise computational efficiency and accuracy. Experimental results demonstrate the effectiveness of our approach, as evidenced by achieving a new state-of-the-art (SOTA) performance on WLASL100 and LSA64. For WLASL100, we achieve a top-1 accuracy of 86.50%, marking a relative improvement of 2.39% over the previous SOTA. For LSA64, we achieve a top-1 accuracy of 99.84%. The artefacts and code related to this study are made publicly online (https://github.com/mpuu00001/Siformer.git).Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- Robustness Evaluation in Hand Pose Estimation Models using Metamorphic Testing2023
- Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation2020
- A Comprehensive Analysis of Synthetic Minority Oversampling Technique (SMOTE) for handling class imbalanceInformation Sciences, 2019
- Actional-Structural Graph Convolutional Networks for Skeleton-Based Action Recognition2019
- SkipNet: Learning Dynamic Routing in Convolutional Networks2018
- A Deep Learning Approach for Analyzing Video and Skeletal Features in Sign Language Recognition2018
- Attention-Based 3D-CNNs for Large-Vocabulary Sign Language RecognitionIEEE Transactions on Circuits and Systems for Video Technology, 2018
- Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN2018
- Benchmarking and Error Diagnosis in Multi-instance Pose Estimation2017
- Chinese sign language recognition based on video sequence appearance modeling2010