Sign2Pose: A Pose-Based Approach for Gloss Prediction Using a Transformer Model
Open Access
Abstract
Word-level sign language recognition (WSLR) is the backbone for continuous sign language recognition (CSLR) that infers glosses from sign videos. Finding the relevant gloss from the sign sequence and detecting explicit boundaries of the glosses from sign videos is a persistent challenge. In this paper, we propose a systematic approach for gloss prediction in WLSR using the Sign2Pose Gloss prediction transformer model. The primary goal of this work is to enhance WLSR’s gloss prediction accuracy with reduced time and computational overhead. The proposed approach uses hand-crafted features rather than automated feature extraction, which is computationally expensive and less accurate. A modified key frame extraction technique is proposed that uses histogram difference and Euclidean distance metrics to select and drop redundant frames. To enhance the model’s generalization ability, pose vector augmentation using perspective transformation along with joint angle rotation is performed. Further, for normalization, we employed YOLOv3 (You Only Look Once) to detect the signing space and track the hand gestures of the signers in the frames. The proposed model experiments on WLASL datasets achieved the top 1% recognition accuracy of 80.9% in WLASL100 and 64.21% in WLASL300. The performance of the proposed model surpasses state-of-the-art approaches. The integration of key frame extraction, augmentation, and pose estimation improved the performance of the proposed gloss prediction model by increasing the model’s precision in locating minor variations in their body posture. We observed that introducing YOLOv3 improved gloss prediction accuracy and helped prevent model overfitting. Overall, the proposed model showed 17% improved performance in the WLASL 100 dataset.Keywords
Funding Information
- JSPS KAKENHI (JP21H03496, JP22K12157, JPMJPR1934)
- JST, PRESTO (JP21H03496, JP22K12157, JPMJPR1934)
This publication has 57 references indexed in Scilit:
- Coupled HMM-based multi-sensor data fusion for sign language recognitionPattern Recognition Letters, 2017
- Deep Hand: How to Train a CNN on 1 Million Hand Images When Your Data is Continuous and Weakly Labelled2016
- Handwritten mathematical symbols datasetData in Brief, 2016
- ASL-LEX: A lexical database of American Sign LanguageBehavior Research Methods, 2016
- Deep Dynamic Neural Networks for Multimodal Gesture Segmentation and RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signersComputer Vision and Image Understanding, 2015
- Joint action recognition and pose estimation from video2015
- Non-manual grammatical marker recognition based on multi-scale, spatio-temporal analysis of head pose and facial expressionsImage and Vision Computing, 2014
- Automatic and Efficient Human Pose Estimation for Sign Language VideosInternational Journal of Computer Vision, 2013
- A boosted classifier tree for hand shape detection2004