MetaPortrait: Identity-Preserving Talking Head Generation with Fast Personalized Adaptation
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 22096-22105
Abstract
In this work, we propose an ID-preserving talking head generation framework, which advances previous methods in two aspects. First, as opposed to interpolating from sparse flow, we claim that dense landmarks are crucial to achieving accurate geometry-aware flow fields. Second, inspired by face-swapping methods, we adaptively fuse the source identity during synthesis, so that the network better preserves the key characteristics of the image portrait. Although the proposed model surpasses prior generation fidelity on established benchmarks, personalized fine-tuning is still needed to further make the talking head generation qualified for real usage. However, this process is rather computationally demanding that is unaffordable to standard users. To alleviate this, we propose a fast adaptation model using a metalearning approach. The learned model can be adapted to a high-quality personalized model as fast as 30 seconds. Last but not least, a spatial-temporal enhancement module is proposed to improve the fine details while ensuring temporal coherency. Extensive experiments prove the significant superiority of our approach over the state of the arts in both one-shot and personalized settings.Keywords
All Related Versions
This publication has 31 references indexed in Scilit:
- Text-based editing of talking-head videoACM Transactions on Graphics, 2019
- The uncanny valley: No need for any further judgments when an avatar looks eerieComputers in Human Behavior, 2019
- Warp-guided GANs for single-photo facial animationACM Transactions on Graphics, 2018
- VoxCeleb2: Deep Speaker Recognition2018
- Deep video portraitsACM Transactions on Graphics, 2018
- Feature-wise transformationsDistill, 2018
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization2017
- A 3D Morphable Model Learnt from 10,000 Faces2016
- A morphable model for the synthesis of 3D faces1999