Emotion Dependent Domain Adaptation for Speech Driven Affective Facial Feature Synthesis
- 14 July 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Affective Computing
- Vol. 13 (3) , 1501-1513
Abstract
Although speech driven facial animation has been studied extensively in the literature, works focusing on the affective content of the speech are limited. This is mostly due to the scarcity of affective audio-visual data. In this article, we improve the affective facial animation using domain adaptation by partially reducing the data scarcity. We first define a domain adaptation to map affective and neutral speech representations to a common latent space in which cross-domain bias is smaller. Then the domain adaptation is used to augment affective representations for each emotion category, including angry, disgust, fear, happy, sad, surprise, and neutral, so that we can better train emotion-dependent deep audio-to-visual (A2V) mapping models. Based on the emotion-dependent deep A2V models, the proposed affective facial synthesis system is realized in two stages: first, speech emotion recognition extracts soft emotion category likelihoods for the utterances; then a soft fusion of the emotion-dependent A2V mapping outputs form the affective facial synthesis. Experimental evaluations are performed on the SAVEE audio-visual dataset. The proposed models are assessed with objective and subjective evaluations. The proposed affective A2V system achieves significant MSE loss improvements in comparison to the recent literature. Furthermore, the resulting facial animations of the proposed system are preferred over the baseline animations in the subjective evaluations.Keywords
Funding Information
- Türkiye Bilimsel ve Teknolojik Araştırma Kurumu (217E107)
This publication has 37 references indexed in Scilit:
- Photo-real talking head with deep bidirectional LSTM2015
- Learning Salient Features for Speech Emotion Recognition Using Convolutional Neural NetworksIEEE Transactions on Multimedia, 2014
- Convolutional Neural Networks for Speech RecognitionIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2014
- Autoencoder-based Unsupervised Domain Adaptation for Speech Emotion RecognitionIEEE Signal Processing Letters, 2014
- Multimodal Emotion Recognition2010
- Statistical mapping between articulatory movements and acoustic spectrum using a Gaussian mixture modelSpeech Communication, 2008
- A coupled HMM approach to video-realistic speech animationPattern Recognition, 2007
- Avatars in Clinical Psychology: A Framework for the Clinical Use of Virtual HumansCyberPsychology & Behavior, 2003
- Quantitative association of vocal-tract and facial behaviorSpeech Communication, 1998
- Virtual human representation and communication in VLNetIEEE Computer Graphics and Applications, 1997