Hybrid Transformers With Attention-Guided Spatial Embeddings for Makeup Transfer and Removal
- 11 September 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34 (4) , 2876-2890
Abstract
Existing makeup transfer methods typically transfer simple makeup colors in a well-conditioned face image and fail to handle makeup style details (e.g., complicated colors and shapes) and facial occlusion. To address these problems, this paper proposes Hybrid Transformers with Attention-guided Spatial Embeddings (named HT-ASE) for makeup transfer and removal. Specifically, a makeup context extractor adopts makeup context global-local interactions to aggregate the high-level context and low-level detail features of the makeup styles, which obtains the context-aware makeup features that encode the complicated colors and shapes of the makeup styles. A face identity extractor adopts a face identity local interaction to aggregate the identity-relevant features of shallow layers into identity semantic features, which refines the identity features. A spatially similarity-aware fusion network introduces a spatially-adaptive layer-instance normalization with attention-guided spatial embeddings to perform semantic alignment and fusion between the makeup and identity features, yielding precise and robust transfer results even with large spatial misalignment and facial occlusion. Extensive experimental results demonstrate that the proposed method outperforms the state-of-the-art methods, especially in the preservation of makeup style details and handling facial occlusion.Keywords
Funding Information
- National Natural Science Foundation of China (62272134)
- Taishan Scholars Program of Shandong Province (tsqn201812106)
This publication has 42 references indexed in Scilit:
- RamGAN: Region Attentive Morphing GAN for Region-Level Makeup Transfer2022
- Spatially-invariant Style-codes Controlled Makeup Transfer2021
- FM2u-Net: Face Morphological Multi-Branch Network for Makeup-Invariant Face Verification2020
- PSGAN: Pose and Expression Robust Spatial-Aware GAN for Customizable Makeup Transfer2020
- Disentangled Representation Learning of Makeup Portraits in the WildInternational Journal of Computer Vision, 2019
- ArcFace: Additive Angular Margin Loss for Deep Face Recognition2019
- BeautyGlow: On-Demand Makeup Transfer Framework With Reversible Generative Network2019
- Diverse Image-to-Image Translation via Disentangled Representations2018
- BiSeNet: Bilateral Segmentation Network for Real-Time Semantic Segmentation2018
- SCA-CNN: Spatial and Channel-Wise Attention in Convolutional Networks for Image Captioning2017