Spatial–Temporal Enhanced Network for Continuous Sign Language Recognition
- 18 July 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34 (3) , 1684-1695
Abstract
Continuous Sign language Recognition (CSLR) aims to generate gloss sequences based on untrimmed sign videos. Since discriminative visual features are essential for CSLR, current efforts mainly focus on strengthening the feature extractor. The feature extractor can be disassembled into a spatial representation module and a short-term temporal module for spatial and visual features modeling. However, existing methods always regard it as a monoblock and rarely implement specific refinements for such two distinct modules, which is difficult to achieve effective modeling of spatial appearance information and temporal motion information. To address the above issues, we proposed a spatial temporal enhanced network which contains a spatial-visual alignment (SVA) module and a temporal feature difference (TFD) module. Specifically, the SVA module conducts an auxiliary task between the spatial features and target gloss sequences to enhance the extraction of hand and facial expressions. Meanwhile, the TFD module is constructed to exploit the underlying dynamic between consecutive frames and inject the aggregated motion information into spatial features to assist short-term temporal modeling. Extensive experimental results demonstrate the effectiveness of the proposed modules and our network achieves state-of-the-art or competitive performance on four public CSLR datasets.Keywords
This publication has 61 references indexed in Scilit:
- Multi-scale local-temporal similarity fusion for continuous sign language recognitionPattern Recognition, 2022
- Deep Sign: Enabling Robust Statistical Continuous Sign Language Recognition via Hybrid CNN-HMMsInternational Journal of Computer Vision, 2018
- Neural Sign Language Translation2018
- SphereFace: Deep Hypersphere Embedding for Face Recognition2017
- Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization2017
- Re-Sign: Re-Aligned End-to-End Sequence Modelling with Deep Recurrent CNN-HMMs2017
- Temporal Segment Networks: Towards Good Practices for Deep Action Recognition2016
- Deep Residual Learning for Image Recognition2016
- Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signersComputer Vision and Image Understanding, 2015
- A Chinese sign language recognition system based on SOFM/SRN/HMMPattern Recognition, 2004