Face Swapping Consistency Transfer with Neural Identity Carrier
Open Access
- 22 November 2021
- journal article
- research article
- Published by MDPI AG in Future Internet
- Vol. 13 (11) , 298
Abstract
Deepfake aims to swap a face of an image with someone else’s likeness in a reasonable manner. Existing methods usually perform deepfake frame by frame, thus ignoring video consistency and producing incoherent results. To address such a problem, we propose a novel framework Neural Identity Carrier (NICe), which learns identity transformation from an arbitrary face-swapping proxy via a U-. By modeling the incoherence between frames as noise, NICe naturally suppresses its disturbance and preserves primary identity information. Concretely, NICe inputs the original frame and learns transformation supervised by swapped pseudo labels. As the temporal incoherence has an uncertain or stochastic pattern, NICe can filter out such outliers and well maintain the target content by uncertainty prediction. With the predicted temporally stable appearance, NICe enhances its details by constraining 3D geometry consistency, making NICe learn fine-grained facial structure across the poses. In this way, NICe guarantees the temporal stableness of deepfake approaches and predicts detailed results against over-smoothness. Extensive experiments on benchmarks demonstrate that NICe significantly improves the quality of existing deepfake methods on video-level. Besides, data generated by our methods can benefit video-level deepfake detection methods.
Keywords
Funding Information
- National Natural Science Foundation of China (62002334, U20B2047)
- Anhui Science Foundation of China (2008085QF296)
- Exploration Fund Project of University of Science and Technology of China (YD3480002001)
- Fundamental Research Funds for Central Universities (WK2100000011)
This publication has 23 references indexed in Scilit:
- OpenFace 2.0: Facial Behavior Analysis Toolkit2018
- Learning a model of facial shape and expression from 4D scansACM Transactions on Graphics, 2017
- Coherent Online Video Style Transfer2017
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Creating a Photoreal Digital Actor: The Digital Emily Project2009
- Aleatory or epistemic? Does it matter?Structural Safety, 2009
- Face swappingACM Transactions on Graphics, 2008
- Heteroscedastic Gaussian process regression2005
- Exchanging Faces in ImagesComputer Graphics Forum, 2004
- Estimating the mean and variance of the target probability distribution1994