MultiFacet: A Multi-Tasking Framework for Speech-to-Sign Language Generation
Open Access
- 9 October 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 205-213
Abstract
Sign language is a rich form of communication, uniquely conveying meaning through a combination of gestures, facial expressions, and body movements. Existing research in sign language generation has predominantly focused on text-to-sign pose generation, while speech-to-sign pose generation remains relatively underexplored. Speech-to-sign language generation models can facilitate effective communication between the deaf and hearing communities. In this paper, we propose an architecture that utilises prosodic information from speech audio and semantic context from text to generate sign pose sequences. In our approach, we adopt a multi-tasking strategy that involves an additional task of predicting Facial Action Units (FAUs). FAUs capture the intricate facial muscle movements that play a crucial role in conveying specific facial expressions during sign language generation. We train our models on an existing Indian Sign language dataset that contains sign language videos with audio and text translations. To evaluate our models, we report Dynamic Time Warping (DTW) and Probability of Correct Keypoints (PCK) scores. We find that combining prosody and text as input, along with incorporating facial action unit prediction as an additional task, outperforms previous models in both DTW and PCK scores. We also discuss the challenges and limitations of speech-to-sign pose generation models to encourage future research in this domain. We release our models, results and code to foster reproducibility and encourage future research1.Keywords
This publication has 24 references indexed in Scilit:
- Speech gesture generation from the trimodal context of text, audio, and speaker identityACM Transactions on Graphics, 2020
- Progressive Transformers for End-to-End Sign Language Production2020
- Gesticulator: A framework for semantically-aware speech-driven gesture generation2020
- Text2Sign: Towards Sign Language Production Using Neural Machine Translation and Generative Adversarial NetworksInternational Journal of Computer Vision, 2020
- LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech2019
- Neural Sign Language Translation2018
- 2D Human Pose Estimation: New Benchmark and State of the Art Analysis2014
- Nonmanuals: their grammatical and prosodic roles2012
- Visual Intonation in the Prosody of a Sign LanguageLanguage and Speech, 2009
- Robust Estimation of a Location ParameterThe Annals of Mathematical Statistics, 1964