DiffusionAvatars: Deferred Diffusion for High-fidelity 3D Head Avatars
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 5481-5492
Abstract
DiffusionAvatars synthesizes a high-fidelity 3D head avatar of a person, offering intuitive control over both pose and expression. We propose a diffusion-based neural ren-derer that leverages generic 2D priors to produce com-pelling images of faces. For coarse guidance of the expression and head pose, we render a neural parametric head model (NPHM) from the target viewpoint, which acts as a proxy geometry of the person. Additionally, to enhance the modeling of intricate facial expressions, we condition DiffusionAvatars directly on the expression codes obtained from NPHM via cross-attention. Finally, to synthesize consistent surface details across different viewpoints and ex-pressions, we rig learnable spatial features to the head's surface via TriPlane lookup in NPHM's canonical space. We train DiffusionAvatars on RGB videos and corresponding fitted NPHM meshes of a person and test the obtained avatars in both self-reenactment and animation scenarios. Our experiments demonstrate that DiffusionAvatars gener-ates temporally consistent and visually appealing videos for novel poses and expressions of a person, outperforming ex-isting approaches.Keywords
All Related Versions
This publication has 67 references indexed in Scilit:
- Control3Diff: Learning Controllable 3D Diffusion Models from Single-view Images2024
- Realistic One-Shot Mesh-Based Head Avatars2022
- Pixel-in-Pixel Net: Towards Efficient Facial Landmark Detection in the WildInternational Journal of Computer Vision, 2021
- FaceScape: A Large-Scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction2020
- ArcFace: Additive Angular Margin Loss for Deep Face Recognition2019
- A Style-Based Generator Architecture for Generative Adversarial Networks2019
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- Pixelwise View Selection for Unstructured Multi-View Stereo2016
- Structure-from-Motion Revisited2016
- A 3D Face Model for Pose and Illumination Invariant Face Recognition2009