MoMask: Generative Masked Modeling of 3D Human Motions
Authors:
Top Cited Papers
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1900-1910
Abstract
We introduce MoMask, a novel masked modeling framework for text-driven 3D human motion generation. In Mo-Mask, a hierarchical quantization scheme is employed to represent human motion as multi-layer discrete motion tokens with high-fidelity details. Starting at the base layer, with a sequence of motion tokens obtained by vector quan-tization, the residual tokens of increasing orders are de-rived and stored at the subsequent layers of the hierar-chy. This is consequently followed by two distinct bidirectional transformers. For the base-layer motion tokens, a Masked Transformer is designated to predict randomly masked motion tokens conditioned on text input at training stage. During generation (i. e. inference) stage, starting from an empty sequence, our Masked Transformer iteratively fills up the missing tokens; Subsequently, a Residual Transformer learns to progressively predict the next-layer tokens based on the results from current layer. Extensive experiments demonstrate that MoMask outperforms the state-of-art methods on the text-to-motion generation task, with an FID of 0.045 (vs e.g. 0.141 of T2M-GPT) on the HumanML3D dataset, and 0.228 (vs 0.514) on KIT-ML, respectively. MoMask can also be seamlessly applied in related tasks without further model fine-tuning, such as text-guided temporal inpainting.Keywords
All Related Versions
This publication has 38 references indexed in Scilit:
- PoseGPT: Quantization-Based 3D Human Motion Generation and Forecasting2022
- TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts2022
- Implicit Neural Representations for Variable Length Human Motion Generation2022
- MotionCLIP: Exposing Human Motion Generation to CLIP Space2022
- TEMOS: Generating Diverse Human Motions from Textual Descriptions2022
- Action2video: Generating Videos of Human 3D ActionsInternational Journal of Computer Vision, 2022
- Language2Pose: Natural Language Grounded Pose Forecasting2019
- Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networksRobotics and Autonomous Systems, 2018
- Deep Video Generation, Prediction and Completion of Human Action Sequences2018
- The KIT Motion-Language DatasetBig Data, 2016