Human Appearance Transfer
Authors:
- Mihai ZanfirMZMihai Zanfir
- Alin-Ionut PopaAPAlin-Ionut Popa
- Andrei ZanfirAZAndrei Zanfir
- Cristian SminchisescuCSCristian Sminchisescu
- 1 June 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 5391-5399
Abstract
We propose an automatic person-to-person appearance transfer model based on explicit parametric 3d human representations and learned, constrained deep translation network architectures for photographic image synthesis. Given a single source image and a single target image, each corresponding to different human subjects, wearing different clothing and in different poses, our goal is to photo-realistically transfer the appearance from the source image onto the target image while preserving the target shape and clothing segmentation layout. Our solution to this new problem is formulated in terms of a computational pipeline that combines (1) 3d human pose and body shape estimation from monocular images, (2) identifying 3d surface colors elements (mesh triangles) visible in both images, that can be transferred directly using barycentric procedures, and (3) predicting surface appearance missing in the first image but visible in the second one using deep learning-based image synthesis techniques. Our model achieves promising results as supported by a perceptual user study where the participants rated around 65% of our results as good, very good or perfect, as well in automated tests (Inception scores and a Faster-RCNN human detector responding very similarly to real and model generated images). We further show how the proposed architecture can be profiled to automatically generate images of a person dressed with different clothing transferred from a person in another image, opening paths for applications in entertainment and photo-editing (e.g. embodying and posing as friends or famous actors), the fashion industry, or affordable online shopping of clothing.Keywords
This publication has 25 references indexed in Scilit:
- ClothCapACM Transactions on Graphics, 2017
- Synthesizing Training Images for Boosting Human 3D Pose Estimation2016
- DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations2016
- Human Parsing with Contextualized Convolutional Neural Network2015
- Where to Buy It: Matching Street Clothing Photos in Online Shops2015
- DeepPose: Human Pose Estimation via Deep Neural Networks2014
- Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural EnvironmentsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Adaptive anisotropic remeshing for cloth simulationACM Transactions on Graphics, 2012
- Articulated people detection and pose estimation: Reshaping the future2012
- Efficient simulation of inextensible clothACM Transactions on Graphics, 2007