Image-to-Video Generation via 3D Facial Dynamics
- 24 May 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 32 (4) , 1805-1819
Abstract
We present a versatile model, FaceAnime, for various video generation tasks from still images. Video generation from a single face image is an interesting problem and usually tackled by utilizing Generative Adversarial Networks (GANs) to integrate information from the input face image and a sequence of sparse facial landmarks. However, the generated face images usually suffer from quality loss, image distortion, identity change, and expression mismatching due to the weak representation capacity of the facial landmarks. In this paper, we propose to “imagine” a face video from a single face image according to the reconstructed 3D face dynamics, aiming to generate a realistic and identity-preserving face video, with precisely predicted pose and facial expression. The 3D dynamics reveal changes of the facial expression and motion, and can serve as a strong prior knowledge for guiding highly realistic face video generation. In particular, we explore face video prediction and exploit a well-designed 3D dynamic prediction network to predict a 3D dynamic sequence for a single face image. The 3D dynamics are then further rendered by the sparse texture mapping algorithm to recover structural details and sparse textures for generating face frames. Our model is versatile for various AR/VR and entertainment applications, such as face video retargeting and face video prediction. Superior experimental results have well demonstrated its effectiveness in generating high-fidelity, identity-preserving, and visually pleasant face video clips from a single source face image.Keywords
All Related Versions
Funding Information
- Open Fund Project of Key Laboratory of Flight Technology and Flight Safety of CAFUC (FZ2020KF10)
- National Science Foundation of China (62006244)
- Project of Comprehensive Reform of Electronic Information Engineering Specialty for Civil Aircraft Maintenance (14002600100017J172)
- Project of Civil Aviation Flight University of China (J2018-56, CJ2019-01, J2020-060)
- National Key Research and Development Program of China (2018AAA0103203)
- Sichuan University Failure Mechanics and Engineering Disaster Prevention and Mitigation Key Laboratory of Sichuan Province Open Foundation (2020FMSCU02)
This publication has 49 references indexed in Scilit:
- Video Summarization with Long Short-Term Memory2016
- Face2Face: Real-Time Face Capture and Reenactment of RGB Videos2016
- Guiding the Long-Short Term Memory Model for Image Caption Generation2015
- Hierarchical recurrent neural network for skeleton based action recognition2015
- FaceWarehouse: A 3D Facial Expression Database for Visual ComputingIEEE Transactions on Visualization and Computer Graphics, 2013
- Expression flow for 3D-aware face component transfer2011
- Automatically Detecting Pain in Video Through Facial Action UnitsIEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 2010
- Weight, Sex, and Facial Expressions: On the Manipulation of Attributes in Generative 3D Face ModelsLecture Notes in Computer Science, 2009
- A morphable model for the synthesis of 3D faces1999
- Long Short-Term MemoryNeural Computation, 1997