Takin-ADA: Emotion Controllable Audio-Driven Animation with Canonical and Landmark Loss Optimization
Preprint
Abstract
Existing audio-driven facial animation methods face critical challenges, including expression leakage, ineffective subtle expression transfer, and imprecise audio-driven synchronization. We discovered that these issues stem from limitations in motion representation and the lack of fine-grained control over facial expressions. To address these problems, we present Takin-ADA, a novel two-stage approach for real-time audio-driven portrait animation. In the first stage, we introduce a specialized loss function that enhances subtle expression transfer while reducing unwanted expression leakage. The second stage utilizes an advanced audio processing technique to improve lip-sync accuracy. Our method not only generates precise lip movements but also allows flexible control over facial expressions and head motions. Takin-ADA achieves high-resolution (512x512) facial animations at up to 42 FPS on an RTX 4090 GPU, outperforming existing commercial solutions. Extensive experiments demonstrate that our model significantly surpasses previous methods in video quality, facial dynamics realism, and natural head movements, setting a new benchmark in the field of audio-driven facial animation.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- EMO: Emote Portrait Alive Generating Expressive Portrait Videos with Audio2Video Diffusion Model Under Weak Conditions2024
- EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesis2024
- Speech driven video editing via an audio-conditioned diffusion modelImage and Vision Computing, 2024
- DaGAN++: Depth-Aware Generative Adversarial Network for Talking Head Video GenerationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2023
- Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priors2023
- Identity-Preserving Talking Face Generation with Landmark and Appearance Priors2023
- StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN2022
- EMOCA: Emotion Driven Monocular Face Capture and Animation2022
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden UnitsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004