Identity-Aware Variational Autoencoder for Face Swapping
- 4 January 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34 (7) , 5466-5479
Abstract
Face swapping aims to transfer the identity of a source face to a target face image while preserving the target attributes (e.g., facial expression, head pose, illumination, and background). Most existing methods use a face recognition model to extract global features from the source face and directly fuse them with the target to generate a swapping result. However, identity-irrelevant attributes (e.g., hairstyle and facial appearances) contribute a lot to the recognition task, and thus swapping this task-specific feature inevitably interfuses source attributes with target ones. In this paper, we propose an identity-aware variational autoencoder (ID-VAE) based face swapping framework, dubbed VAFSwap, which learns disentangled identity and attribute representations for high-fidelity face swapping. In particular, we overcome the unpaired training barrier of VAE and impose a proxy identity on the latent space by exploiting the weak supervision from an auxiliary image set whose identity is averaged from multiple collected face images. To explicitly guide the identity fusion, we further devise an identity-associated matrix that corresponds different face regions with their identity representations to perform identity-related feature interactions. Finally, we incorporate spatial dimensions into the latent space and exploit the generative priors of a pre-trained face generator, allowing the effective elimination of noticeable swapping artifacts. Extensive experiments on the FaceForensics++ and CelebA-HQ datasets demonstrate that our method outperforms the state-of-the-art significantly.Keywords
This publication has 55 references indexed in Scilit:
- Designing One Unified Framework for High-Fidelity Face Reenactment and Swapping2022
- StyleSwap: Style-Based Generator Empowers Robust Face Swapping2022
- Average faces: How does the averaging process change faces physically and perceptually?Cognition, 2021
- Disentangling Factors of Variation with Cycle-Consistent Variational Auto-encoders2018
- Nonlinear 3D Face Morphable Model2018
- CosFace: Large Margin Cosine Loss for Deep Face Recognition2018
- Face2Face: Real-Time Face Capture and Reenactment of RGB Videos2016
- The MegaFace Benchmark: 1 Million Faces for Recognition at Scale2016
- Photorealistic Face De-Identification by Aggregating Donors’ Face Components2015
- Creating a Photoreal Digital Actor: The Digital Emily Project2009