A Music-Driven Deep Generative Adversarial Model for Guzheng Playing Animation
- 28 September 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 29 (2) , 1400-1414
Abstract
To date relatively few efforts have been made on the automatic generation of musical instrument playing animations. This problem is challenging due to the intrinsically complex, temporal relationship between music and human motion as well as the lacking of high quality music-playing motion datasets. In this article, we propose a fully automatic, deep learning based framework to synthesize realistic upper body animations based on novel guzheng music input. Specifically, based on a recorded audiovisual motion capture dataset, we delicately design a generative adversarial network (GAN) based approach to capture the temporal relationship between the music and the human motion data. In this process, data augmentation is employed to improve the generalization of our approach to handle a variety of guzheng music inputs. Through extensive objective and subjective experiments, we show that our method can generate visually plausible guzheng-playing animations that are well synchronized with the input guzheng music, and it can significantly outperform the state-of-the-art methods. In addition, through an ablation study, we validate the contributions of the carefully-designed modules in our framework.Keywords
This publication has 64 references indexed in Scilit:
- Exploring relationships between pianists’ body movements, their expressive intentions, and structural elements of the musicMusicae Scientiae, 2011
- ELAN: A Software Package for Analysis and Visualization of MEG, EEG, and LFP SignalsComputational Intelligence and Neuroscience, 2011
- Mann‐Whitney U Test2010
- Dancing‐to‐Music Character AnimationComputer Graphics Forum, 2006
- Natural head motion synthesis driven by acoustic prosodic featuresComputer Animation and Virtual Worlds, 2005
- Long Short-Term MemoryNeural Computation, 1997
- Visual Perception of Performance Manner in the Movements of Solo MusiciansPsychology of Music, 1993
- Algorithms for the Longest Common Subsequence ProblemJournal of the ACM, 1977
- Optimal error bounds for cubic spline interpolationJournal of Approximation Theory, 1976
- Minimum prediction residual principle applied to speech recognitionIEEE Transactions on Acoustics, Speech, and Signal Processing, 1975