IFaceUV: Intuitive Motion Facial Image Generation by Identity Preservation via UV map
Authors:
Preprint
Abstract
Reenacting facial images is an important task that can find numerous applications. We proposed IFaceUV, a fully differentiable pipeline that properly combines 2D and 3D information to conduct the facial reenactment task. The three-dimensional morphable face models (3DMMs) and corresponding UV maps are utilized to intuitively control facial motions and textures, respectively. Two-dimensional techniques based on 2D image warping is further required to compensate for missing components of the 3DMMs such as backgrounds, ear, hair and etc. In our pipeline, we first extract 3DMM parameters and corresponding UV maps from source and target images. Then, initial UV maps are refined by the UV map refinement network and it is rendered to the image with the motion manipulated 3DMM parameters. In parallel, we warp the source image according to the 2D flow field obtained from the 2D warping network. Rendered and warped images are combined in the final editing network to generate the final reenactment image. Additionally, we tested our model for the audio-driven facial reenactment task. Extensive qualitative and quantitative experiments illustrate the remarkable performance of our method compared to other state-of-the-art methods.Keywords
All Related Versions
This publication has 39 references indexed in Scilit:
- What Matters in Unsupervised Optical Flow2020
- Neural Voice Puppetry: Audio-Driven Facial Reenactment2020
- Synthesizing Coupled 3D Face Modalities by Trunk-Branch Generative Adversarial Networks2020
- Fast Bi-Layer Neural Synthesis of One-Shot Realistic Head Avatars2020
- Realistic Speech-Driven Facial Animation with GANsInternational Journal of Computer Vision, 2019
- ReenactGAN: Learning to Reenact Faces via Boundary Transfer2018
- Recycle-GAN: Unsupervised Video Retargeting2018
- X2Face: A Network for Controlling Face Generation Using Images, Audio, and Pose Codes2018
- Gaussian mixture 3D morphable face modelPattern Recognition, 2018
- Perceptual Losses for Real-Time Style Transfer and Super-Resolution2016