Diffusion Video Autoencoders: Toward Temporally Consistent Face Video Editing via Disentangled Video Encoding
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 6091-6100
Abstract
Inspired by the impressive performance of recent face image editing methods, several studies have been naturally proposed to extend these methods to the face video editing task. One of the main challenges here is temporal consistency among edited frames, which is still unresolved. To this end, we propose a novel face video editing framework based on diffusion autoencoders that can successfully extract the decomposed features - for the first time as a face video editing model - of identity and motion from a given video. This modeling allows us to edit the video by simply manipulating the temporally invariant feature to the desired direction for the consistency. Another unique strength of our model is that, since our model is based on diffusion models, it can satisfy both reconstruction and edit capabilities at the same time, and is robust to corner cases in wild face videos (e.g. occluded faces) unlike the existing GAN-based methods. 1 1 Project page: https://diff-video-ae.github.ioKeywords
All Related Versions
Funding Information
- Institute of Information & communications Technology Planning & Evaluation (IITP) (2019-0-00075)
- Artificial Intelligence Graduate School Program(KAIST) (2021-0-02068)
- National Research Foundation of Korea (NRF) (2018RlA5Al059921)
This publication has 19 references indexed in Scilit:
- BiSeNet V2: Bilateral Network with Guided Aggregation for Real-Time Semantic SegmentationInternational Journal of Computer Vision, 2021
- Designing an encoder for StyleGAN image manipulationACM Transactions on Graphics, 2021
- One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing2021
- Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation2021
- Closed-Form Factorization of Latent Semantics in GANs2021
- Face identity disentanglement via latent space mappingACM Transactions on Graphics, 2020
- ArcFace: Additive Angular Margin Loss for Deep Face Recognition2019
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- Multiscale structural similarity for image quality assessment2004
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004