Vid2speech: Speech reconstruction from silent video
Authors:
- 1 March 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 5095-5099
Abstract
Speechreading is a notoriously difficult task for humans to perform. In this paper we present an end-to-end model based on a convolutional neural network (CNN) for generating an intelligible acoustic speech signal from silent video frames of a speaking person. The proposed CNN generates sound features for each frame based on its neighboring frames. Waveforms are then synthesized from the learned speech features to produce intelligible speech. We show that by leveraging the automatic feature learning capabilities of a CNN, we can obtain state-of-the-art word intelligibility on the GRID dataset, and show promising results for learning out-of-vocabulary (OOV) words.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Lipreading with long short-term memory2016
- Decoding visemes: Improving machine lip-reading2016
- Statistical conversion of silent articulation into audible speech using full-covariance HMMComputer Speech & Language, 2016
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification2015
- A review of recent advances in visual speech decodingImage and Vision Computing, 2014
- An audio-visual corpus for speech perception and automatic speech recognitionThe Journal of the Acoustical Society of America, 2006
- A neural network model of the articulatory-acoustic forward mapping trained on recordings of articulatory parametersThe Journal of the Acoustical Society of America, 2004
- Extraction of visual features for lipreadingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2002
- Line spectrum representation of linear predictor coefficients of speech signalsThe Journal of the Acoustical Society of America, 1975
- Acoustic Theory of Speech Production1971