Learning from What is Already Out There: Few-shot Sign Language Recognition with Online Dictionaries
Authors:
- 5 January 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-6
Abstract
Today's sign language recognition models require large training corpora of laboratory-like videos, whose collection involves an extensive workforce and financial resources. As a result, only a handful of such systems are publicly available, not to mention their limited localization capabilities for less-populated sign languages. Utilizing online text-to-video dictionaries, which inherently hold annotated data of various attributes and sign languages, and training models in a few-shot fashion hence poses a promising path for the democratization of this technology. In this work, we collect and open-source the UWB-SL-Wild few-shot dataset, the first of its kind training resource consisting of dictionary-scraped videos. This dataset represents the actual distribution and characteristics of available online sign language data. We select glosses that directly overlap with the already existing datasets WLASL100 and ASLLVD and share their class mappings to allow for transfer learning experiments. Apart from providing baseline results on a pose-based architecture, we introduce a novel approach to training sign language recognition models in a few-shot scenario, resulting in state-of-the-art results on ASLLVD-Skeleton and ASLLVD-Skeleton-20 datasets with top- 1 accuracy of 30.97 % and 95.45 %, respectively.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity FieldsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2019
- Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language VideosIEEE Transactions on Pattern Analysis and Machine Intelligence, 2019
- Deep convolutional neural networks for sign language recognition2018
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization2017
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Deep Sign: Hybrid CNN-HMM for Continuous Sign Language Recognition2016
- Purdue RVL-SLLL ASL database for automatic recognition of American Sign Language2003
- Real-time American Sign Language recognition from video using hidden Markov models2002
- Real-time American sign language recognition using desk and wearable computer based videoIEEE Transactions on Pattern Analysis and Machine Intelligence, 1998