Diffusion‐based Human Motion Style Transfer with Semantic Guidance
- 9 October 2024
- journal article
- research article
- Published by Wiley in Computer Graphics Forum
- Vol. 43 (8)
Abstract
3D Human motion style transfer is a fundamental problem in computer graphic and animation processing. Existing AdaIN‐based methods necessitate datasets with balanced style distribution and content/style labels to train the clustered latent space. However, we may encounter a single unseen style example in practical scenarios, but not in sufficient quantity to constitute a style cluster for AdaIN‐based methods. Therefore, in this paper, we propose a novel two‐stage framework for few‐shot style transfer learning based on the diffusion model. Specifically, in the first stage, we pre‐train a diffusion‐based text‐to‐motion model as a generative prior so that it can cope with various content motion inputs. In the second stage, based on the single style example, we fine‐tune the pre‐trained diffusion model in a few‐shot manner to make it capable of style transfer. The key idea is regarding the reverse process of diffusion as a motion‐style translation process since the motion styles can be viewed as special motion variations. During the fine‐tuning for style transfer, a simple yet effective semantic‐guided style transfer loss coordinated with style example reconstruction loss is introduced to supervise the style transfer in CLIP semantic space. The qualitative and quantitative evaluations demonstrate that our method can achieve state‐of‐the‐art performance and has practical applications. The source code is available at https://github.com/hlcdyy/diffusion-based-motion-style-transfer.Keywords
All Related Versions
Funding Information
- National Key Research and Development Program of China (2022YFB3303202)
This publication has 41 references indexed in Scilit:
- EDGE: Editable Dance Generation From Music2023
- Generating Human Motion from Textual Descriptions with Discrete Representations2023
- TEACH: Temporal Action Composition for 3D Humans2022
- Generating Diverse and Natural 3D Human Motions from Text2022
- DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation2022
- Contact-Aware Retargeting of Skinned Motion2021
- HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation2021
- Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization2017
- Image Style Transfer Using Convolutional Neural Networks2016
- Generalized time warping for multi-modal alignment of human motion2012