Unsupervised High-Resolution Portrait Gaze Correction and Animation
- 27 July 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 31 (10577149) , 5272-5286
Abstract
This paper proposes a gaze correction and animation method for high-resolution, unconstrained portrait images, which can be trained without the gaze angle and the head pose annotations. Common gaze-correction methods usually require annotating training data with precise gaze, and head pose information. Solving this problem using an unsupervised method remains an open problem, especially for high-resolution face images in the wild, which are not easy to annotate with gaze and head pose labels. To address this issue, we first create two new portrait datasets: CelebGaze ( $256 \times 256$ ) and high-resolution CelebHQGaze ( $512 \times 512$ ). Second, we formulate the gaze correction task as an image inpainting problem, addressed using a Gaze Correction Module (GCM) and a Gaze Animation Module (GAM). Moreover, we propose an unsupervised training strategy, i.e., Synthesis-As-Training, to learn the correlation between the eye region features and the gaze angle. As a result, we can use the learned latent space for gaze animation with semantic interpolation in this space. Moreover, to alleviate both the memory and the computational costs in the training and the inference stage, we propose a Coarse-to-Fine Module (CFM) integrated with GCM and GAM. Extensive experiments validate the effectiveness of our method for both the gaze correction and the gaze animation tasks in both low and high-resolution face datasets in the wild and demonstrate the superiority of our method with respect to the state of the art.Keywords
All Related Versions
Funding Information
- PRIN Project CREATIVE (2020ZSL9F9)
- EU H2020 Project AI4Media (951911)
- EU H2020 Project SPRING (871245)
This publication has 53 references indexed in Scilit:
- Deep Learning Face Attributes in the Wild2015
- Learning to look up: Realtime monocular gaze correction using machine learning2015
- Gaze locking2013
- Gaze correction for home video conferencingACM Transactions on Graphics, 2012
- PatchMatchACM Transactions on Graphics, 2009
- Example-Based Rendering of Eye MovementsComputer Graphics Forum, 2009
- Multiscale structural similarity for image quality assessment2004
- Gaze manipulation for one-to-one teleconferencing2003
- Eye Gaze Correction with Stereovision for Video-Teleconferencing2002
- Multiparty videoconferencing at virtual social distance1994