SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognition
Authors:
Top Cited Papers
- 1 October 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 11067-11076
Abstract
Hand gesture serves as a critical role in sign language. Current deep-learning-based sign language recognition (SLR) methods may suffer insufficient interpretability and overfitting due to limited sign data sources. In this paper, we introduce the first self-supervised pre-trainable SignBERT with incorporated hand prior for SLR. Sign-BERT views the hand pose as a visual token, which is derived from an off-the-shelf pose extractor. The visual tokens are then embedded with gesture state, temporal and hand chirality information. To take full advantage of available sign data sources, SignBERT first performs self-supervised pre-training by masking and reconstructing visual tokens. Jointly with several mask modeling strategies, we attempt to incorporate hand prior in a model-aware method to better model hierarchical context over the hand sequence. Then with the prediction head added, SignBERT is fine-tuned to perform the downstream SLR task. To validate the effectiveness of our method on SLR, we perform extensive experiments on four public benchmark datasets, i.e., NMFs-CSL, SLR500, MSASL and WLASL. Experiment results demonstrate the effectiveness of both self-supervised learning and imported hand prior. Furthermore, we achieve state-of-the-art performance on all benchmarks with a notable gain.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China
- Youth Innovation Promotion Association
This publication has 37 references indexed in Scilit:
- Sphere-meshes for real-time hand modeling and trackingACM Transactions on Graphics, 2016
- Capturing Hands in Action Using Discriminative Salient Points and Physics SimulationInternational Journal of Computer Vision, 2016
- A Real-Time Hand Posture Recognition System Using Deep Neural NetworksACM Transactions on Intelligent Systems and Technology, 2015
- Realtime and Robust Hand Tracking from Depth2014
- Evolutionary Quasi-Random Search for Hand Articulations Tracking2014
- Glove: Global Vectors for Word Representation2014
- Interactive Markerless Articulated Hand Motion Tracking Using RGB and Depth Data2013
- On Space-Time Interest PointsInternational Journal of Computer Vision, 2005
- Spherical blend skinning2005
- Pose space deformation2000