Transflower

Authors:
Top Cited Papers
Open Access
Publisher website
Google Scholar
Abstract
Dance requires skillful composition of complex movements that follow rhythmic, tonal and timbral features of music. Formally, generating dance conditioned on a piece of music can be expressed as a problem of modelling a high-dimensional continuous motion signal, conditioned on an audio signal. In this work we make two contributions to tackle this problem. First, we present a novel probabilistic autoregressive architecture that models the distribution over future poses with a normalizing flow conditioned on previous poses as well as music context, using a multimodal transformer encoder. Second, we introduce the currently largest 3D dance-motion dataset, obtained with a variety of motion-capture technologies, and including both professional and casual dancers. Using this dataset, we compare our new model against two baselines, via objective metrics and a user study, and show that both the ability to model a probability distribution, as well as being able to attend over a large motion and music context are necessary to produce interesting, diverse, and realistic dance that matches the music.
Funding Information
  • The Swedish Research Council a.k.a. Vetenskapsrådet (grants no. 2018-05409 and 2019-03694)
  • the Knut and Alice Wallenberg Foundation (WASP)
  • the Marianne and Marcus Wallenberg Foundation (MMW 2020.0102)
  • GENCI (allocation 2020-[A0091011996])

This publication has 43 references indexed in Scilit: