Taming Diffusion Models for Music-driven Conducting Motion Generation
Preprint
Abstract
Generating the motion of orchestral conductors from a given piece of symphony music is a challenging task since it requires a model to learn semantic music features and capture the underlying distribution of real conducting motion. Prior works have applied Generative Adversarial Networks (GAN) to this task, but the promising diffusion model, which recently showed its advantages in terms of both training stability and output quality, has not been exploited in this context. This paper presents Diffusion-Conductor, a novel DDIM-based approach for music-driven conducting motion generation, which integrates the diffusion model to a two-stage learning framework. We further propose a random masking strategy to improve the feature robustness, and use a pair of geometric loss functions to impose additional regularizations and increase motion diversity. We also design several novel metrics, including Frechet Gesture Distance (FGD) and Beat Consistency Score (BC) for a more comprehensive evaluation of the generated motion. Experimental results demonstrate the advantages of our model.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- MotionDiffuse: Text-Driven Human Motion Generation With Diffusion ModelIEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
- Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation2023
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Self-Supervised Music Motion Synchronization Learning for Music-Driven Conducting Motion GenerationJournal of Computer Science and Technology, 2022
- Speech gesture generation from the trimodal context of text, audio, and speaker identityACM Transactions on Graphics, 2020
- DeepDance: Music-to-Dance Motion Choreography With Adversarial LearningIEEE Transactions on Multimedia, 2020