One-Shot Human Motion Transfer via Occlusion-Robust Flow Prediction and Neural Texturing
- 27 January 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 3939-3952
Abstract
Human motion transfer aims at animating a static source image with a driving video. While recent advances in one-shot human motion transfer have led to significant improvement in results, it remains challenging for methods with 2D body landmarks, skeleton and semantic mask to accurately capture correspondences between source and driving poses due to the large variation in motion and articulation complexity. In addition, the accuracy and precision of DensePose degrade the image quality for neural-rendering-based methods. To address the limitations and by both considering the importance of appearance and geometry for motion transfer, in this work, we proposed a unified framework that combines multi-scale feature warping and neural texture mapping to recover better 2D appearance and 2.5D geometry, partly by exploiting the information from DensePose, yet adapting to its inherent limited accuracy. Our model takes advantage of multiple modalities by jointly training and fusing them, which allows it to robust neural texture features that cope with geometric errors as well as multi-scale dense motion flow that better preserves appearance. Experimental results with full and half-view body video datasets demonstrate that our model can generalize well and achieve competitive results, and that it is particularly effective in handling challenging cases such as those with substantial self-occlusions.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62302104)
- Natural Science Foundation of Guangdong Province (2023A1515012884)
- Science and Technology Program of Guangzhou (SL2023A04J01625, RIE2020)
- Industry Alignment Fund – Industry Collaboration Projects
- Singapore Telecommunications Limited
- Singtel Cognitive and Artificial Intelligence Lab for Enterprises
This publication has 50 references indexed in Scilit:
- Motion Transformer for Unsupervised Image Animation2022
- HumanGAN: A Generative Model of Human Images2021
- Neural Re-rendering of Humans from a Single Image2020
- Voxceleb: Large-scale speaker verification in the wildComputer Speech & Language, 2020
- Human Motion Transfer from Poses in the Wild2020
- Dense Pose Transfer2018
- Deformable GANs for Pose-Based Human Image Generation2018
- FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks2017
- Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields2017
- Deep Residual Learning for Image Recognition2016