D-LORD for Motion Stylization
- 3 December 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Systems, Man, and Cybernetics: Systems
- Vol. 55 (2) , 1374-1387
Abstract
This article introduces a novel framework named double-latent optimization for representation disentanglement (D-LORD), which is designed for motion stylization (motion style transfer and motion retargeting). The primary objective of this framework is to separate the class and content information from a given motion sequence using a data-driven latent optimization approach. Here, class refers to person-specific style, such as a particular emotion or an individual’s identity, while content relates to the style-agnostic aspect of an action, such as walking or jumping, as universally understood concepts. The key advantage of D-LORD is its ability to perform style transfer without needing paired motion data. Instead, it utilizes class and content labels during the latent optimization process. By disentangling the representation, the framework enables the transformation of one motion sequence’s style to another’s style using adaptive instance normalization. The proposed D-LORD framework is designed with a focus on generalization, allowing it to handle different class and content labels for various applications. In addition, it can generate diverse motion sequences when specific class and content labels are provided. The framework’s efficacy is demonstrated through experimentation on three datasets: 1) the CMU XIA dataset for motion style transfer; 2) the multimodal human action database dataset; and 3) the RRIS Ability dataset for motion retargeting. Notably, this article presents the first generalized framework for motion style transfer and motion retargeting, showcasing its potential contributions in this area.Keywords
All Related Versions
Funding Information
- Agency for Science, Technology and Research (A*STAR) under the National Robotics Programme (NRP) BAU Grant, through Assistive Robotics Programme (M22NBK0074)
This publication has 37 references indexed in Scilit:
- Fast human motion transfer based on a meta networkInformation Sciences, 2021
- An Asian-centric human movement database capturing activities of daily livingScientific Data, 2020
- A Style-Based Generator Architecture for Generative Adversarial Networks2019
- Multimodal Unsupervised Image-to-Image Translation2018
- Diverse Image-to-Image Translation via Disentangled Representations2018
- Neural Kinematic Networks for Unsupervised Motion Retargetting2018
- StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation2018
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- Neural Style Transfer via Meta Networks2018
- Image Style Transfer Using Convolutional Neural Networks2016