Sign-MExD: An Expert-Infused Diffusion Model for Sign Language Production

Authors:

Abstract
Sign language production (SLP) aims to generate semantically aligned videos from textual statements, where the conversion from textual glosses to sign poses is a crucial step. In the field of SLP, the state-of-the-art is based on diffusion models that contain hundreds of millions of parameters and require numerous denoising steps to generate sign poses. Consequently, a major limitation is their high computational cost, resulting in slow training and inference speeds as the models become large. To address this challenge, we explore a mixture-of-experts (MoE) architecture that can scale model capacity while reducing computational cost for training and inference. We propose Sign-MExD, an expert-infused model for SLP that combines the strengths of the MoE mechanism with a diffusion model. Specifically, Sign-MExD learns to adaptively optimize the computational resources allocated to understanding input gloss sequences and generating the respective sign pose sequences, enabling heterogeneous computation aligned with varying gloss-to-pose complexities. Experiments on the PHOENIX14T and How2Sign datasets demonstrate that Sign-MExD offers competitive performance compared to the state-of-the-art and achieves significantly higher ROUGE scores.

This publication has 26 references indexed in Scilit: