SignAligner: Harmonizing Complementary Pose Modalities for Coherent Sign Language Generation
Preprint
Abstract
Sign language generation aims to produce diverse sign representations based on spoken language. However, achieving realistic and naturalistic generation remains a significant challenge due to the complexity of sign language, which encompasses intricate hand gestures, facial expressions, and body movements. In this work, we introduce PHOENIX14T+, an extended version of the widely-used RWTH-PHOENIX-Weather 2014T dataset, featuring three new sign representations: Pose, Hamer and Smplerx. We also propose a novel method, SignAligner, for realistic sign language generation, consisting of three stages: text-driven pose modalities co-generation, online collaborative correction of multimodality, and realistic sign video synthesis. First, by incorporating text semantics, we design a joint sign language generator to simultaneously produce posture coordinates, gesture actions, and body movements. The text encoder, based on a Transformer architecture, extracts semantic features, while a cross-modal attention mechanism integrates these features to generate diverse sign language representations, ensuring accurate mapping and controlling the diversity of modal features. Next, online collaborative correction is introduced to refine the generated pose modalities using a dynamic loss weighting strategy and cross-modal attention, facilitating the complementarity of information across modalities, eliminating spatiotemporal conflicts, and ensuring semantic coherence and action consistency. Finally, the corrected pose modalities are fed into a pre-trained video generation network to produce high-fidelity sign language videos. Extensive experiments demonstrate that SignAligner significantly improves both the accuracy and expressiveness of the generated sign videos.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Gloss-driven Conditional Diffusion Models for Sign Language ProductionACM Transactions on Multimedia Computing, Communications, and Applications, 2025
- T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Text2024
- A Survey of Knowledge Enhanced Pre-Trained Language ModelsIEEE Transactions on Knowledge and Data Engineering, 2023
- BERTTune: Fine-Tuning Neural Machine Translation with BERTScore2021
- Skeleton-based Chinese sign language recognition and generation for bidirectional communication between deaf and hearing peopleNeural Networks, 2020
- Learning Three-dimensional Skeleton Data from Sign Language VideoACM Transactions on Intelligent Systems and Technology, 2020
- Text2Sign: Towards Sign Language Production Using Neural Machine Translation and Generative Adversarial NetworksInternational Journal of Computer Vision, 2020
- Deep Gesture Video Generation With Learning on Regions of InterestIEEE Transactions on Multimedia, 2019
- HamNoSys to SiGML Conversion System for Sign Language AutomationProcedia Computer Science, 2016
- Automatic Translation System to Spanish Sign Language with a Virtual Interpreter2009