Text-to-Korean Sign Language Pose Sequence Generation Using Non-Manual Signal Conditioning and Multi-Scale Temporal Refinement
Open Access
Abstract
Automatic sign language generation has the potential to support information accessibility for deaf and hard-of-hearing individuals. Generating sign language pose sequences from natural language text can serve as an intermediate representation for avatar-based sign language expression and sign language video synthesis. However, text-to-sign pose generation is challenging because sign language conveys meaning through both manual movements and non-manual signals, while requiring temporally coherent motion over local and sentence-level contexts. In addition, text length does not directly correspond to the number of pose frames required for sign language expression. To address these issues, this study proposes a text-to-Korean Sign Language (KSL) pose generation model based on non-manual signal conditioning and multi-scale temporal refinement. The proposed framework integrates a text encoder, pose decoder, non-manual signal conditioning, multi-scale temporal refinement, and length prediction/blending. The model generates normalized 58-joint KSL keypoint sequences from morpheme-level text inputs and jointly optimizes pose reconstruction, motion continuity, bone consistency, PCK-aware precision, non-manual signal prediction, and length consistency. Experimental results on a KSL text–pose dataset show that the proposed model outperforms text-only and Transformer-based baselines. Compared with the Transformer text-to-pose baseline, the proposed model reduced MPJPE from 0.408236 to 0.316366 and Pose MAE from 0.165473 to 0.128570. It also improved PCK@0.05 from 0.136090 to 0.163928 and reduced the length relative error from 0.221455 to 0.127152. In particular, the best-threshold non-manual F1 substantially increased from 0.010859 to 0.494566. These results suggest that text-based KSL pose generation should jointly consider non-manual expressions, length consistency, and long-term temporal motion structure rather than relying only on frame-wise keypoint prediction. However, the reported improvements should be interpreted as coordinate- and label-level evidence, not as a complete validation of linguistic meaningfulness or real-world accessibility.Keywords
Funding Information
- Institute for Information and Communications Technology Promotion (RS-2025-02216275)
This publication has 35 references indexed in Scilit:
- SignGen: End-to-End Sign Language Video Generation with Latent Diffusion2024
- A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars2024
- Pose-Guided Fine-Grained Sign Language Video Generation2024
- Semantic-driven diffusion for sign language production with gloss-pose latent spaces alignmentComputer Vision and Image Understanding, 2024
- MotionCLIP: Exposing Human Motion Generation to CLIP Space2022
- TEMOS: Generating Diverse Human Motions from Textual Descriptions2022
- Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers and Mixture Density NetworksInternational Journal of Computer Vision, 2021
- Progressive Transformers for End-to-End Sign Language Production2020
- Text2Sign: Towards Sign Language Production Using Neural Machine Translation and Generative Adversarial NetworksInternational Journal of Computer Vision, 2020
- Grammar, Gesture, and Meaning in American Sign Language2003