Towards Automatic Translation of Syrian Arabic Sign Language: A Preliminary Dataset and Transformer Based Baseline
Authors:
- Yumna Al ShalakYSYumna Al Shalak
- Omar HamdounOHOmar Hamdoun
- Nada GhneimNGNada Ghneim
Preprint
- 15 September 2026
- preprint
- Published by Elsevier in SSRN Electronic Journal
Abstract
Accurate communication between deaf and hearing individuals remains challenging because qualified sign-language interpreters are not always available, especially for under-resourced sign languages such as Syrian Arabic Sign Language. This paper presents a newly developedpreliminary dataset for Syrian Arabic Sign Language and establishes a baseline end-to-end system for translating continuous sign language videos into grammatically correct written Arabic. The proposed system provides an initial benchmark for video-to-text sign language translation and serves as a foundation for future research on Syrian Arabic Sign Language translation. The task is formulated as video-to-text sign language translation. To capture fine-grained signing cues, including hand configurations, motion dynamics, and facial expressions, we combine features extracted from two pretrained vision models and model their temporaldependencies with a Transformer encoder. An AraBART decoder then generates Arabic sentences conditioned on the encoded video representation. To improve signer invariance, the proposed system incorporates adversarial training through a Gradient Reversal Layer (GRL), while backbone-specific preprocessing is applied to better accommodate the input characteristics of the two visual encoders. The system is evaluated on the newly collected dataset under a signer-dependent setting. The best configuration, incorporating GRL and backbone-specific preprocessing, achieves a WER of 0.166. These results indicate that combining pretrained visual representations, adversarial training, backbone-specificpreprocessing, and AraBART-based text generation is a promising direction for Syrian Arabic Sign Language translation under controlled conditions. However, performance drops substantially in the signer independent setting, indicating that cross-signer generalization remains challenging.Keywords
This publication has 5 references indexed in Scilit:
- SignStreamNet: Streaming Sign Language Video-to-Text Translation for Accessibility2025
- Hand Gesture Recognition for Multi-Culture Sign Language Using Graph and General Deep Learning NetworkIEEE Open Journal of the Computer Society, 2024
- Two-Stage Deep Learning Solution for Continuous Arabic Sign Language Recognition Using Word Count Prediction and Motion ImagesIEEE Access, 2023
- ArabSign: A Multi-modality Dataset and Benchmark for Continuous Arabic Sign Language Recognition2023
- Utilizing motion and spatial features for sign language gesture recognition using cascaded CNN and LSTM modelsTurkish Journal of Electrical Engineering and Computer Sciences, 2022