EmoSpeaker: One-shot Fine-grained Emotion-Controlled Talking Face Generation
Authors:
- Guanwen FengGFGuanwen Feng
- Haoran ChengHCHaoran Cheng
- Yunan LiYLYunan Li
- Zhiyuan MaZMZhiyuan Ma
- Chaoneng LiCLChaoneng Li
- Zhihao QianZQZhihao Qian
- Qiguang MiaoQiguang Miao
- Chi-Man PunChi-Man Pun
Preprint
Abstract
Implementing fine-grained emotion control is crucial for emotion generation tasks because it enhances the expressive capability of the generative model, allowing it to accurately and comprehensively capture and express various nuanced emotional states, thereby improving the emotional quality and personalization of generated content. Generating fine-grained facial animations that accurately portray emotional expressions using only a portrait and an audio recording presents a challenge. In order to address this challenge, we propose a visual attribute-guided audio decoupler. This enables the obtention of content vectors solely related to the audio content, enhancing the stability of subsequent lip movement coefficient predictions. To achieve more precise emotional expression, we introduce a fine-grained emotion coefficient prediction module. Additionally, we propose an emotion intensity control method using a fine-grained emotion matrix. Through these, effective control over emotional expression in the generated videos and finer classification of emotion intensity are accomplished. Subsequently, a series of 3DMM coefficient generation networks are designed to predict 3D coefficients, followed by the utilization of a rendering network to generate the final video. Our experimental results demonstrate that our proposed method, EmoSpeaker, outperforms existing emotional talking face generation methods in terms of expression variation and lip synchronization. Project page: https://peterfanfan.github.io/EmoSpeaker/Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Emotion-Controllable Generalized Talking Face Generation2022
- Speech2Talking-Face: Inferring and Driving a Face with Synchronized Audio-Visual Representation2021
- Speech-Driven Expressive Talking Lips with Conditional Sequential Generative Adversarial NetworksIEEE Transactions on Affective Computing, 2019
- The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American EnglishPLOS ONE, 2018
- Audio-driven facial animation by joint end-to-end learning of pose and emotionACM Transactions on Graphics, 2017
- CREMA-D: Crowd-Sourced Emotional Multimodal Actors DatasetIEEE Transactions on Affective Computing, 2014
- A No-Reference Image Blur Metric Based on the Cumulative Probability of Blur Detection (CPBD)IEEE Transactions on Image Processing, 2011