Audio-Driven Facial Animation with Deep Learning: A Survey
Open Access
- 28 October 2024
- journal article
- review article
- Published by MDPI AG in Information
- Vol. 15 (11) , 675
Abstract
Audio-driven facial animation is a rapidly evolving field that aims to generate realistic facial expressions and lip movements synchronized with a given audio input. This survey provides a comprehensive review of deep learning techniques applied to audio-driven facial animation, with a focus on both audio-driven facial image animation and audio-driven facial mesh animation. These approaches employ deep learning to map audio inputs directly onto 3D facial meshes or 2D images, enabling the creation of highly realistic and synchronized animations. This survey also explores evaluation metrics, available datasets, and the challenges that remain, such as disentangling lip synchronization and emotions, generalization across speakers, and dataset limitations. Lastly, we discuss future directions, including multi-modal integration, personalized models, and facial attribute modification in animations, all of which are critical for the continued development and application of this technology.Keywords
Funding Information
- European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie (900025)
This publication has 111 references indexed in Scilit:
- CelebV-HQ: A Large-Scale Video Facial Attributes Dataset2022
- StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN2022
- Talking-Head Generation with Rhythmic Head Motion2020
- Speech-Driven Facial Animation Using Cascaded GANs for Learning of Motion and Texture2020
- Audio‐driven emotional speech animation for interactive virtual charactersComputer Animation and Virtual Worlds, 2019
- Lip Movements Generation at a Glance2018
- X2Face: A Network for Controlling Face Generation Using Images, Audio, and Pose Codes2018
- Lip Reading in the Wild2017
- Speech-Driven Facial Animation Using a Shared Gaussian Process Latent Variable ModelLecture Notes in Computer Science, 2009
- HMM-Based Audio Keyword Generation2004