G2P-DDM: Generating Sign Pose Sequence from Gloss Sequence with Discrete Diffusion Model
Open Access
- 24 March 2024
- journal article
- research article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 38 (6) , 6234-6242
Abstract
The Sign Language Production (SLP) project aims to automatically translate spoken languages into sign sequences. Our approach focuses on the transformation of sign gloss sequences into their corresponding sign pose sequences (G2P). In this paper, we present a novel solution for this task by converting the continuous pose space generation problem into a discrete sequence generation problem. We introduce the Pose-VQVAE framework, which combines Variational Autoencoders (VAEs) with vector quantization to produce a discrete latent representation for continuous pose sequences. Additionally, we propose the G2P-DDM model, a discrete denoising diffusion architecture for length-varied discrete sequence data, to model the latent prior. To further enhance the quality of pose sequence generation in the discrete space, we present the CodeUnet model to leverage spatial-temporal information. Lastly, we develop a heuristic sequential clustering method to predict variable lengths of pose sequences for corresponding gloss sequences. Our results show that our model outperforms state-of-the-art G2P models on the public SLP evaluation benchmark. For more generated results, please visit our project page: https://slpdiffusier.github.io/g2p-ddm.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformer2022
- Hierarchical Transformers Are More Efficient Language Models2022
- PiSLTRc: Position-Informed Sign Language Transformer With Content-Aware ConvolutionIEEE Transactions on Multimedia, 2021
- Spatial-Temporal Multi-Cue Network for Sign Language Recognition and TranslationIEEE Transactions on Multimedia, 2021
- Progressive Transformers for End-to-End Sign Language Production2020
- Skeleton-based Chinese sign language recognition and generation for bidirectional communication between deaf and hearing peopleNeural Networks, 2020
- OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity FieldsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2019
- Study on density peaks clustering based on k-nearest neighbors and principal component analysisKnowledge-Based Systems, 2016
- A Learning Algorithm for Continually Running Fully Recurrent Neural NetworksNeural Computation, 1989