VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Prior
Authors:
Top Cited Papers
- 25 March 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 713-722
Abstract
Audio-driven talking head generation has drawn much attention in recent years, and many efforts have been made in lip-sync, facial motion, head pose generation, and video quality. However, no model has yet led or tied on all these metrics due to the one-to-many mapping between audio and motion. In this paper, we propose VividTalk, a two-stage generic framework that supports generating high-visual quality talking head videos with all the above properties. Specifically, in the first stage, we map the audio to mesh by learning two motions, including non-rigid facial motion and rigid head motion. For facial motion, both blendshape and vertex are adopted as the intermediate representation to maximize the representation ability of the model. For head motion, a novel learnable head pose codebook with a two-phase training mechanism is proposed. In the second stage, we proposed a dual branch motion-vae and a generator to transform the meshes into dense motion and synthesize high-quality video frame-by-frame. Extensive experiments show that the proposed VividTalk can generate high-visual quality talking head videos with lip-sync and realistic enhanced by a large margin, and outperforms previous state-of-the-art works in objective and subjective comparisons. The code will be publicly released upon publication.Keywords
All Related Versions
This publication has 32 references indexed in Scilit:
- Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation2021
- One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing2021
- Flow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset2021
- Talking-Head Generation with Rhythmic Head Motion2020
- Neural Voice Puppetry: Audio-Driven Facial Reenactment2020
- DAVD-Net: Deep Audio-Aided Video Decompression of Talking Heads2020
- Realistic Speech-Driven Facial Animation with GANsInternational Journal of Computer Vision, 2019
- Lip Movements Generation at a Glance2018
- Out of Time: Automated Lip Sync in the Wild2017
- Face2Face: Real-Time Face Capture and Reenactment of RGB Videos2016