Categorical Codebook Matching for Embodied Character Controllers
Open Access
- 19 July 2024
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 43 (4) , 1-14
Abstract
Translating motions from a real user onto a virtual embodied avatar is a key challenge for character animation in the metaverse. In this work, we present a novel generative framework that enables mapping from a set of sparse sensor signals to a full body avatar motion in real-time while faithfully preserving the motion context of the user. In contrast to existing techniques that require training a motion prior and its mapping from control to motion separately, our framework is able to learn the motion manifold as well as how to sample from it at the same time in an end-to-end manner. To achieve that, we introduce a technique called codebook matching which matches the probability distribution between two categorical codebooks for the inputs and outputs for synthesizing the character motions. We demonstrate this technique can successfully handle ambiguity in motion generation and produce high quality character controllers from unstructured motion capture data. Our method is especially useful for interactive applications like virtual reality or video games where high accuracy and responsiveness are needed.Keywords
This publication has 45 references indexed in Scilit:
- QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation2023
- Generating Human Motion from Textual Descriptions with Discrete Representations2023
- Combining Motion Matching and Orientation Prediction to Animate Avatars for Consumer‐Grade VR DevicesComputer Graphics Forum, 2022
- AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensing2022
- Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensors2022
- FLAG: Flow-based 3D Avatar Generation from Sparse Observations2022
- MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement2021
- Full-Body Motion from a Single Head-Mounted Device: Generating SMPL Poses from Partial Observations2021
- Action-Conditioned 3D Human Motion Synthesis with Transformer VAE2021
- Precomputing avatar behavior from human motion dataProceedings of the 2004 ACM SIGGRAPH/Eurographics symposium on Computer animation - SCA '04, 2004