A Novel Approach for Visual Speech Recognition Using the Partition-Time Masking and Swin Transformer 3D Convolutional Model
Open Access
Abstract
Visual speech recognition is a technology that relies on visual information, offering unique advantages in noisy environments or when communicating with individuals with speech impairments. However, this technology still faces challenges, such as limited generalization ability due to different speech habits, high recognition error rates caused by confusable phonemes, and difficulties adapting to complex lighting conditions and facial occlusions. This paper proposes a lip reading data augmentation method—Partition-Time Masking (PTM)—to address these challenges and improve lip reading models’ performance and generalization ability. Applying nonlinear transformations to the training data enhances the model’s generalization ability when handling diverse speakers and environmental conditions. A lip-reading recognition model architecture, Swin Transformer and 3D Convolution (ST3D), was designed to overcome the limitations of traditional lip-reading models that use ResNet-based front-end feature extraction networks. By adopting a strategy that combines Swin Transformer and 3D convolution, the proposed model enhances performance. To validate the effectiveness of the Partition-Time Masking data augmentation method, experiments were conducted on the LRW video dataset using the DC-TCN model, achieving a peak accuracy of 92.15%. The ST3D model was validated on the LRW and LRW1000 video datasets, achieving a maximum accuracy of 56.1% on the LRW1000 dataset and 91.8% on the LRW dataset, outperforming current mainstream lip reading models and demonstrating superior performance on challenging easily confused samples.Keywords
Funding Information
- National Natural Science Foundation of China (U21A20120, 52175033, LQN25E050013, 6142606243106, YQZX-B202401, XY2024012)
- Zhejiang Provincial Natural Science Foundation of China (U21A20120, 52175033, LQN25E050013, 6142606243106, YQZX-B202401, XY2024012)
- National Key Laboratory of Transient Impact (U21A20120, 52175033, LQN25E050013, 6142606243106, YQZX-B202401, XY2024012)
- Special Instrument Development and Cultivation Project of Zhejiang University (U21A20120, 52175033, LQN25E050013, 6142606243106, YQZX-B202401, XY2024012)
- A Project Supported by Scientific Research Fund of Zhejiang University (U21A20120, 52175033, LQN25E050013, 6142606243106, YQZX-B202401, XY2024012)
This publication has 41 references indexed in Scilit:
- EEG-based emotion recognition with autoencoder feature fusion and MSC-TimesNet modelComputer Methods in Biomechanics and Biomedical Engineering, 2025
- A novel framework using 3D-CNN and BiLSTM model with dynamic learning rate scheduler for visual speech recognitionSignal, Image and Video Processing, 2024
- A Lip Reading Method Based on 3D Convolutional Vision TransformerIEEE Access, 2022
- GridMix: Strong regularization through local context mappingPattern Recognition, 2021
- Speech rehabilitation in post-stroke aphasia using visual illustration of speech articulators: A case report studyClinical Linguistics & Phonetics, 2020
- Pushing the boundaries of audiovisual word recognition using Residual Networks and LSTMsComputer Vision and Image Understanding, 2018
- Facial Landmark Detection: A Literature SurveyInternational Journal of Computer Vision, 2018
- Audiovisual Integration and Lipreading Abilities of Older Adults with Normal and Impaired HearingEar & Hearing, 2007
- Extraction of visual features for lipreadingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2002
- Active appearance modelsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2001