Speaker-Independent Speech-Driven Visual Speech Synthesis using Domain-Adapted Acoustic Models
Preprint
Abstract
Speech-driven visual speech synthesis involves mapping features extracted from acoustic speech to the corresponding lip animation controls for a face model. This mapping can take many forms, but a powerful approach is to use deep neural networks (DNNs). However, a limitation is the lack of synchronized audio, video, and depth data required to reliably train the DNNs, especially for speaker-independent models. In this paper, we investigate adapting an automatic speech recognition (ASR) acoustic model (AM) for the visual speech synthesis problem. We train the AM on ten thousand hours of audio-only data. The AM is then adapted to the visual speech synthesis domain using ninety hours of synchronized audio-visual speech. Using a subjective assessment test, we compared the performance of the AM-initialized DNN to one with a random initialization. The results show that viewers significantly prefer animations generated from the AM-initialized DNN than the ones generated using the randomly initialized model. We conclude that visual speech synthesis can significantly benefit from the powerful representation of speech in the ASR acoustic models.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- Learning Dynamic Stream Weights For Coupled-HMM-based Audio-visual Speech RecognitionIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2015
- HMM trajectory-guided sample selection for photo-realistic talking headMultimedia Tools and Applications, 2014
- Comprehensive many-to-many phoneme-to-viseme mapping and its application for concatenative visual speech synthesisSpeech Communication, 2013
- Realistic Mouth-Synching for Speech-Driven Talking Face Using Articulatory ModellingIEEE Transactions on Multimedia, 2007
- A coupled HMM approach to video-realistic speech animationPattern Recognition, 2007
- Audio/visual mapping with cross-modal hidden Markov modelsIEEE Transactions on Multimedia, 2005
- Speech-driven facial animation with realistic dynamicsIEEE Transactions on Multimedia, 2005
- Hidden Markov Model Inversion for Audio-to-Visual Conversion in an MPEG-4 Facial Animation SystemJournal of Signal Processing Systems, 2001
- Photo-realistic talking-heads from image samplesIEEE Transactions on Multimedia, 2000
- Quantitative association of vocal-tract and facial behaviorSpeech Communication, 1998