Towards View-Independent Viseme Recognition Based on CNNS and Synthetic Data
- 1 October 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 589-593
Abstract
Visual Speech Recognition is the ability to interpret spoken text using video information only. To address such task automatically, recent works have employed Deep Learning and obtained high accuracy on the recognition of words and sentences uttered in controlled environments, with limited head-pose variation. However, the accuracy drops for multi-view datasets and when it comes to interpreting isolated mouth shapes, such as visemes, the values reported are considerably lower, as shorter segments of speech lack temporal and contextual information. In this work, we evaluate the applicability of synthetic datasets for assisting recognition of visemes in real-world data acquired under controlled and uncontrolled environments, using GRID and AVICAR datasets, respectively. We create two large-scale synthetic 2D datasets based on realistic 3D facial models - with near-frontal and multi-view mouth images. We perform experiments that indicate that a transfer learning approach using synthetic data can get higher accuracy than training from scratch using real data only, on both scenarios.Keywords
This publication has 16 references indexed in Scilit:
- Phoneme-to-viseme mappings: the good, the bad, and the uglySpeech Communication, 2017
- Xception: Deep Learning with Depthwise Separable Convolutions2017
- Vid2speech: Speech reconstruction from silent video2017
- Temporal Multimodal Learning in Audiovisual Speech Recognition2016
- Deep Learning of Mouth Shapes for Sign Language2015
- A review of recent advances in visual speech decodingImage and Vision Computing, 2014
- Selecting relevant visual features for speechreading2009
- Lipreading by Locality Discriminant Graph2007
- Visual speech recognition with loosely synchronized feature streams2005
- Extraction of visual features for lipreadingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2002