Combining Residual Networks with LSTMs for Lipreading
Preprint
Abstract
We propose an end-to-end deep learning architecture for word-level visual speech recognition. The system is a combination of spatiotemporal convolutional, residual and bidirectional Long Short-Term Memory networks. We train and evaluate it on the Lipreading In-The-Wild benchmark, a challenging database of 500-size target-words consisting of 1.28sec video excerpts from BBC TV broadcasts. The proposed network attains word accuracy equal to 83.0, yielding 6.8 absolute improvement over the current state-of-the-art, without using information about word boundaries during training or testing.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition2017
- Audio-visual speech recognition using deep learningApplied Intelligence, 2014
- Word embeddings for speech recognition2014
- A review of recent advances in visual speech decodingImage and Vision Computing, 2014
- Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research GroupsIEEE Signal Processing Magazine, 2012
- The Natural Statistics of Audiovisual SpeechPLOS Computational Biology, 2009
- An audio-visual corpus for speech perception and automatic speech recognitionThe Journal of the Acoustical Society of America, 2006
- Recent advances in the automatic recognition of audiovisual speechProceedings of the IEEE, 2003
- Lipreading from color videoIEEE Transactions on Image Processing, 1997
- Confusions Among Visually Perceived ConsonantsJournal of Speech and Hearing Research, 1968