DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder
Open Access
- 26 October 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 4281-4289
Abstract
While recent research has made significant progress in speech-driven talking face generation, the quality of the generated video still lags behind that of real recordings. One reason for this is the use of handcrafted intermediate representations like facial landmarks and 3DMM coefficients, which are designed based on human knowledge and are insufficient to precisely describe facial movements. Additionally, these methods require an external pretrained model for extracting these representations, whose performance sets an upper bound on talking face generation. To address these limitations, we propose a novel method called DAE-Talker that leverages data-driven latent representations obtained from a diffusion autoencoder (DAE). DAE contains an image encoder that encodes an image into a latent vector and a DDIM-based image decoder that reconstructs the image from it. We train our DAE on talking face video frames and then extract their latent representations as the training target for a Conformer-based speech2latent model. During inference, DAE-Talker first predicts the latents from speech and then generates the video frames with the image decoder in DAE from the predicted latents. This allows DAE-Talker to synthesize full video frames and produce natural head movements that align with the content of speech, rather than relying on a predetermined head pose from a template video. We also introduce pose modelling in speech2latent for pose controllability. Additionally, we propose a novel method for generating continuous video frames with the DDIM-based image decoder trained on individual frames, eliminating the need for modelling the joint distribution of consecutive frames directly. Our experiments show that DAE-Talker outperforms existing popular methods in lip-sync, video fidelity, and pose naturalness. We also conduct ablation studies to analyze the effectiveness of the proposed techniques and demonstrate the pose controllability of DAE-Talker.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- MakeltTalkACM Transactions on Graphics, 2020
- A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild2020
- Neural Voice Puppetry: Audio-Driven Facial Reenactment2020
- Realistic Speech-Driven Facial Animation with GANsInternational Journal of Computer Vision, 2019
- LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech2019
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- A deep learning approach for generalized speech animationACM Transactions on Graphics, 2017
- Audio-driven facial animation by joint end-to-end learning of pose and emotionACM Transactions on Graphics, 2017
- Synthesizing ObamaACM Transactions on Graphics, 2017
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004