Fusion of Multimodal Spatio-Temporal Features and 3D Deformable Convolution Based on Sign Language Recognition in Sensor Networks
Open Access
Abstract
Sign language is a complex and dynamic visual language that requires the coordinated movement of various body parts, such as the hands, arms, and limbs—making it an ideal application domain for sensor networks to capture and interpret human gestures accurately. To address the intricate task of precise and expedient SLR from raw videos, this study introduces a novel deep learning approach by devising a multimodal framework for SLR. Specifically, feature extraction models are built based on two modalities: skeleton and RGB images. In this paper, we firstly propose a Multi-Stream Spatio-Temporal Graph Convolutional Network (MSGCN) that relies on three modules: a decoupling graph convolutional network, a self-emphasizing temporal convolutional network, and a spatio-temporal joint attention module. These modules are combined to capture the spatio-temporal information in multi-stream skeleton features. Secondly, we propose a 3D ResNet model based on deformable convolution (D-ResNet) to model complex spatial and temporal sequences in the original raw images. Finally, a gating mechanism-based Multi-Stream Fusion Module (MFM) is employed to merge the results of the two modalities. Extensive experiments are conducted on the public datasets AUTSL and WLASL, achieving competitive results compared to state-of-the-art systems.Keywords
Funding Information
- The National Natural Science Foundation of China (61902199, 61872197, SJCX23_0272)
- Postgraduate Research and Practice Innovation Program of Jiangsu Province (61902199, 61872197, SJCX23_0272)
This publication has 56 references indexed in Scilit:
- ASL-3DCNN: American sign language recognition technique using 3-D convolutional neural networksMultimedia Tools and Applications, 2021
- An Integrated CNN-LSTM Model for Bangla Lexical Sign Language Recognition2020
- Decoupling GCN with DropGraph Module for Skeleton-Based Action Recognition2020
- BSL-1K: Scaling Up Co-articulated Sign Language Recognition Using Mouthing Cues2020
- Hand pose aware multimodal isolated sign language recognitionMultimedia Tools and Applications, 2020
- Score-Level Multi Cue Fusion for Sign Language Recognition2020
- A Two-Stream CNN Framework for American Sign Language Recognition Based on Multimodal Data Fusion2019
- Real-Time Sign Language Gesture (Word) Recognition from Video Sequences Using CNN and RNN2018
- SIFT-Based Arabic Sign Language Recognition System2015
- Kinect-based Taiwanese sign-language recognition systemMultimedia Tools and Applications, 2014