GAC-GAN: A General Method for Appearance-Controllable Human Video Motion Transfer
Authors:
- Dongxu WeiDWDongxu Wei
- Xiaowei XuXXXiaowei Xu
- Haibin ShenHSHaibin Shen
- Kejie HuangKHKejie Huang
Preprint
Abstract
Human video motion transfer has a wide range of applications in multimedia, computer vision and graphics. Recently, due to the rapid development of Generative Adversarial Networks (GANs), there has been significant progress in the field. However, almost all existing GAN-based works are prone to address the mapping from human motions to video scenes, with scene appearances are encoded individually in the trained models. Therefore, each trained model can only generate videos with a specific scene appearance, new models are required to be trained to generate new appearances. Besides, existing works lack the capability of appearance control. For example, users have to provide video records of wearing new clothes or performing in new backgrounds to enable clothes or background changing in their synthetic videos, which greatly limits the application flexibility. In this paper, we propose GAC-GAN, a general method for appearance-controllable human video motion transfer. To enable general-purpose appearance synthesis, we propose to include appearance information in the conditioning inputs. Thus, once trained, our model can generate new appearances by altering the input appearance information. To achieve appearance control, we first obtain the appearance-controllable conditioning inputs and then utilize a two-stage GAC-GAN to generate the corresponding appearance-controllable outputs, where we utilize an ACGAN loss and a shadow extraction module for output foreground and background appearance control respectively. We further build a solo dance dataset containing a large number of dance videos for training and evaluation. Experimental results show that, our proposed GAC-GAN can not only support appearance-controllable human video motion transfer but also achieve higher video quality than state-of-art methods.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Neural Rendering and Reenactment of Human Actor VideosACM Transactions on Graphics, 2019
- Dance Dance Generation: Motion Transfer for Internet Videos2019
- Deep Video‐Based Performance CloningComputer Graphics Forum, 2019
- Look into Person: Joint Body Parsing & Pose Estimation Network and a New BenchmarkIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks2017
- Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network2017
- Real-time motion retargeting to highly varied user-created morphologiesACM Transactions on Graphics, 2008
- A Parametric Texture Model Based on Joint Statistics of Complex Wavelet CoefficientsInternational Journal of Computer Vision, 2000
- A hierarchical approach to interactive motion editing for human-like figures1999