TD3Net: A Temporal Densely Connected Multi-Dilated Convolutional Network for Lipreading
Authors:
Preprint
Abstract
The word-level lipreading approach typically employs a two-stage framework with separate frontend and backend architectures to model dynamic lip movements. Each component has been extensively studied, and in the backend architecture, temporal convolutional networks (TCNs) have been widely adopted in state-of-the-art methods. Recently, dense skip connections have been introduced in TCNs to mitigate the limited density of the receptive field, thereby improving the modeling of complex temporal representations. However, their performance remains constrained owing to potential information loss regarding the continuous nature of lip movements, caused by blind spots in the receptive field. To address this limitation, we propose TD3Net, a temporal densely connected multi-dilated convolutional network that combines dense skip connections and multi-dilated temporal convolutions as the backend architecture. TD3Net covers a wide and dense receptive field without blind spots by applying different dilation factors to skip-connected features. Experimental results on a word-level lipreading task using two large publicly available datasets, Lip Reading in the Wild (LRW) and LRW-1000, indicate that the proposed method achieves performance comparable to state-of-the-art methods. It achieved higher accuracy with fewer parameters and lower floating-point operations compared to existing TCN-based backend architectures. Moreover, visualization results suggest that our approach effectively utilizes diverse temporal features while preserving temporal continuity, presenting notable advantages in lipreading systems. The code is available at our GitHub repository: https://github.com/Leebh-kor/TD3Net-A-Temporal-Densely-Connected-Multi-dilated-Convolutional-Network-for-LipreadingKeywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Audio–visual speech recognition based on regulated transformer and spatio–temporal fusion strategy for driver assistive systemsExpert Systems with Applications, 2024
- A Lip Reading Method Based on 3D Convolutional Vision TransformerIEEE Access, 2022
- A Survey of Research on Lipreading TechnologyIEEE Access, 2020
- Can We Read Speech Beyond the Lips? Rethinking RoI Selection for Deep Visual Speech Recognition2020
- Single image resolution enhancement by efficient dilated densely connected residual networkSignal Processing: Image Communication, 2019
- Pushing the boundaries of audiovisual word recognition using Residual Networks and LSTMsComputer Vision and Image Understanding, 2018
- Learning to lip read words by watching videosComputer Vision and Image Understanding, 2018
- Combining Residual Networks with LSTMs for Lipreading2017
- Lip Reading Sentences in the Wild2017
- An audio-visual corpus for speech perception and automatic speech recognitionThe Journal of the Acoustical Society of America, 2006