Sign-MExD: An Expert-Infused Diffusion Model for Sign Language Production
Authors:
- 22 October 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1200-1205
Abstract
Sign language production (SLP) aims to generate semantically aligned videos from textual statements, where the conversion from textual glosses to sign poses is a crucial step. In the field of SLP, the state-of-the-art is based on diffusion models that contain hundreds of millions of parameters and require numerous denoising steps to generate sign poses. Consequently, a major limitation is their high computational cost, resulting in slow training and inference speeds as the models become large. To address this challenge, we explore a mixture-of-experts (MoE) architecture that can scale model capacity while reducing computational cost for training and inference. We propose Sign-MExD, an expert-infused model for SLP that combines the strengths of the MoE mechanism with a diffusion model. Specifically, Sign-MExD learns to adaptively optimize the computational resources allocated to understanding input gloss sequences and generating the respective sign pose sequences, enabling heterogeneous computation aligned with varying gloss-to-pose complexities. Experiments on the PHOENIX14T and How2Sign datasets demonstrate that Sign-MExD offers competitive performance compared to the state-of-the-art and achieves significantly higher ROUGE scores.Keywords
This publication has 26 references indexed in Scilit:
- Alleviating data insufficiency for Chinese sign language recognitionVisual Intelligence, 2023
- ViGT: proposal-free video grounding with a learnable token in the transformerScience China Information Sciences, 2023
- How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language2021
- Progressive Transformers for End-to-End Sign Language Production2020
- Skeleton-based Chinese sign language recognition and generation for bidirectional communication between deaf and hearing peopleNeural Networks, 2020
- Text2Sign: Towards Sign Language Production Using Neural Machine Translation and Generative Adversarial NetworksInternational Journal of Computer Vision, 2020
- NN-Based Czech Sign Language SynthesisLecture Notes in Computer Science, 2019
- Neural Sign Language Translation2018
- Educational resources and implementation of a Greek sign language synthesis architectureComputers & Education, 2007
- Grammar, Gesture, and Meaning in American Sign Language2003