Flow Guided Transformable Bottleneck Networks for Motion Retargeting
Authors:
- 1 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 10790-10800
Abstract
Human motion retargeting aims to transfer the motion of one person in a "driving" video or set of images to another person. Existing efforts leverage a long training video from each target person to train a subject-specific motion transfer model. However, the scalability of such methods is limited, as each model can only generate videos for the given target subject, and such training videos are labor-intensive to acquire and process. Few-shot motion transfer techniques, which only require one or a few images from a target, have recently drawn considerable attention. Methods addressing this task generally use either 2D or explicit 3D representations to transfer motion, and in doing so, sacrifice either accurate geometric modeling or the flexibility of an end-to-end learned representation. Inspired by the Transformable Bottleneck Network, which renders novel views and manipulations of rigid objects, we propose an approach based on an implicit volumetric representation of the image content, which can then be spatially manipulated using volumetric flow fields. We address the challenging question of how to aggregate information across different body poses, learning flow fields that allow for combining content from the appropriate regions of input images of highly non-rigid human subjects performing complex motions into a single implicit volumetric representation. This allows us to learn our 3D representation solely from videos of moving people. Armed with both 3D object understanding and end-to-end learned rendering, this categorically novel representation delivers state-of-the-art image generation quality, as shown by our quantitative and qualitative evaluations.Keywords
All Related Versions
This publication has 40 references indexed in Scilit:
- Synthesizing Images of Humans in Unseen Poses2018
- Multistage Adversarial Losses for Pose-Based Human Image Synthesis2018
- Unsupervised Person Image Synthesis in Arbitrary Poses2018
- Disentangled Person Image Generation2018
- Deep Residual Learning for Image Recognition2016
- DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations2016
- SMPLACM Transactions on Graphics, 2015
- Interpolating and approximating implicit surfaces from polygon soupACM Transactions on Graphics, 2004
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004
- Reconstruction and representation of 3D objects with radial basis functions2001