Bangla Sign Language Recognition With Multimodal Deep Learning Fusion
Authors:
Open Access
- 27 April 2025
- journal article
- Published by Wiley in Engineering Reports
- Vol. 7 (4)
Abstract
Sign languages are conveyed through hand gestures and body language; different cultures and languages have unique sign language representations. It is difficult for the general population to interpret all these variations. A Bangla sign language recognition system can help mitigate this problem. In Bangladesh, approximately 13.7 million people are affected by hearing impairments, highlighting the importance of enhancing the Bangla sign language recognition system. In this study, a merged dataset consists of 200‐word classes and 7,000 videos performed by 16 signers. A unique aspect of this dataset is that it includes two modalities. Initially, we extracted frames and audio from the videos. We then employed two frameworks to capture body keypoints from the frames. Additionally, we contributed to expanding the dataset by creating a custom dataset that adds more samples per class. During preprocessing, we augmented the training data, improving our model's learning capacity and reducing overfitting. We developed a system leveraging various deep learning techniques, simultaneously working with two different modalities. We proposed a custom CNN‐LSTM model for pose estimation and integrated VGGish and 2D CNN techniques to process audio in our multimodal model. The CNN‐LSTM model attained an accuracy of 87.08% for pose estimation using the OpenPose framework. The applied ViT model achieved the best performance with 88.52% and 88.39% F1 score to predict body keypoints directly. The VGGish technique delivered the best result for the audio dataset, achieving 81.91% accuracy and 80.42% F1 score. Finally, the late fusion approach combining the ViT and VGGish networks employing the multimodal data achieved the best performance, achieving 94.71% accuracy and 94.52% F1 score.Keywords
This publication has 26 references indexed in Scilit:
- TriChronoNet: Advancing electricity price prediction with Multi-module fusionApplied Energy, 2024
- MLMSign: Multi-lingual multi-modal illumination-invariant sign language recognitionIntelligent Systems with Applications, 2024
- D2MNet for music generation joint driven by facial expressions and dance movementsArray, 2024
- Computer vision-based six layered ConvNeural network to recognize sign language for both numeral and alphabet signsBiomimetic Intelligence and Robotics, 2023
- BdSL47: A complete depth-based Bangla sign alphabet and digit datasetData in Brief, 2023
- Systematic review of machine learning methods applied to ecoacoustics and soundscape monitoringHeliyon, 2023
- Real-time computer vision-based gestures recognition system for bangla sign language using multiple linguistic features analysisMultimedia Tools and Applications, 2023
- Affective state prediction of E-learner using SS-ROA based deep LSTMArray, 2023
- Deep Learning-based Bangla Sign Language Detection with an Edge DeviceIntelligent Systems with Applications, 2023
- A hybrid approach for Bangla sign language recognition using deep transfer learning model with random forest classifierExpert Systems with Applications, 2022