A Transformer-Based Framework for Greek Sign Language Production using Extended Skeletal Motion Representations
Authors:
- Chrysa PratikakiCPChrysa Pratikaki
- Panagiotis FilntisisPFPanagiotis Filntisis
- Athanasios KatsamanisAKAthanasios Katsamanis
- Anastasios RoussosARAnastasios Roussos
- Petros MaragosPMPetros Maragos
Preprint
Abstract
Sign Languages are the primary form of communication for Deaf communities across the world. To break the communication barriers between the Deaf and Hard-of-Hearing and the hearing communities, it is imperative to build systems capable of translating the spoken language into sign language and vice versa. Building on insights from previous research, we propose a deep learning model for Sign Language Production (SLP), which to our knowledge is the first attempt on Greek SLP. We tackle this task by utilizing a transformer-based architecture that enables the translation from text input to human pose keypoints, and the opposite. We evaluate the effectiveness of the proposed pipeline on the Greek SL dataset Elementary23, through a series of comparative analyses and ablation studies. Our pipeline's components, which include data-driven gloss generation, training through video to text translation and a scheduling algorithm for teacher forcing - auto-regressive decoding seem to actively enhance the quality of produced SL videos.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- There and Back Again: 3D Sign Language Generation from Text Using Back-Translation2022
- Signing at Scale: Learning to Co-Articulate Signs for Large-Scale Photo-Realistic Sign Language Production2022
- Progressive Transformers for End-to-End Sign Language Production2020
- Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields2017
- Convolutional Pose Machines2016
- Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signersComputer Vision and Image Understanding, 2015
- The Development and Evaluation of a Speech-to-Sign Translation System to Assist TransactionsInternational Journal of Human–Computer Interaction, 2003