DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder
Authors:
- Chenpeng DuCDChenpeng Du
- Qi ChenQCQi Chen
- Tianyu HeTHTianyu He
- Xu TanXTXu Tan
- Xie ChenXCXie Chen
- Kai YuKYKai Yu
- Sheng ZhaoSZSheng Zhao
- Jiang BianJBJiang Bian
Preprint
Abstract
While recent research has made significant progress in speech-driven talking face generation, the quality of the generated video still lags behind that of real recordings. One reason for this is the use of handcrafted intermediate representations like facial landmarks and 3DMM coefficients, which are designed based on human knowledge and are insufficient to precisely describe facial movements. Additionally, these methods require an external pretrained model for extracting these representations, whose performance sets an upper bound on talking face generation. To address these limitations, we propose a novel method called DAE-Talker that leverages data-driven latent representations obtained from a diffusion autoencoder (DAE). DAE contains an image encoder that encodes an image into a latent vector and a DDIM image decoder that reconstructs the image from it. We train our DAE on talking face video frames and then extract their latent representations as the training target for a Conformer-based speech2latent model. This allows DAE-Talker to synthesize full video frames and produce natural head movements that align with the content of speech, rather than relying on a predetermined head pose from a template video. We also introduce pose modelling in speech2latent for pose controllability. Additionally, we propose a novel method for generating continuous video frames with the DDIM image decoder trained on individual frames, eliminating the need for modelling the joint distribution of consecutive frames directly. Our experiments show that DAE-Talker outperforms existing popular methods in lip-sync, video fidelity, and pose naturalness. We also conduct ablation studies to analyze the effectiveness of the proposed techniques and demonstrate the pose controllability of DAE-Talker.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation2024
- Diffusion Video Autoencoders: Toward Temporally Consistent Face Video Editing via Disentangled Video Encoding2023
- Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesis2022
- AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis2021
- LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalization2021
- Realistic Speech-Driven Facial Animation with GANsInternational Journal of Computer Vision, 2019
- A deep learning approach for generalized speech animationACM Transactions on Graphics, 2017
- Audio-driven facial animation by joint end-to-end learning of pose and emotionACM Transactions on Graphics, 2017
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004