Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control
- 5 September 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 27 (1) , 14-28
Abstract
This paper introduces a new generative deep learning network for human motion synthesis and control. Our key idea is to combine recurrent neural networks (RNNs) and adversarial training for human motion modeling. We first describe an efficient method for training an RNN model from prerecorded motion data. We implement RNNs with long short-term memory (LSTM) cells because they are capable of addressing the nonlinear dynamics and long term temporal dependencies present in human motions. Next, we train a refiner network using an adversarial loss, similar to generative adversarial networks (GANs), such that refined motion sequences are indistinguishable from real mocap data using a discriminative network. The resulting model is appealing for motion synthesis and control because it is compact, contact-aware, and can generate an infinite number of naturally looking motions with infinite lengths. Our experiments show that motions generated by our deep learning model are always highly realistic and comparable to high-quality motion capture data. We demonstrate the power and effectiveness of our models by exploring a variety of applications, ranging from random motion synthesis, online/offline motion control, and motion filtering. We show the superiority of our generative model by comparison against baseline models.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (61772499)
- Natural Science Foundation of Beijing Municipality (L182052)
This publication has 25 references indexed in Scilit:
- Realtime style transfer for unlabeled heterogeneous human motionACM Transactions on Graphics, 2015
- Motion graphs++ACM Transactions on Graphics, 2012
- Physically valid statistical models for human motion generationACM Transactions on Graphics, 2011
- Interactive generation of human animation with deformable motion modelsACM Transactions on Graphics, 2009
- Modeling spatial and temporal variation in motion dataACM Transactions on Graphics, 2009
- Constraint-based motion optimization using a statistical dynamic modelACM Transactions on Graphics, 2007
- Motion textureACM Transactions on Graphics, 2002
- Long Short-Term MemoryNeural Computation, 1997
- On the limited memory BFGS method for large scale optimizationMathematical Programming, 1989
- Generalization of backpropagation with application to a recurrent gas market modelNeural Networks, 1988