Stratified Avatar Generation from Sparse Observations
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 153-163
Abstract
Estimating 3D full-body avatars from AR/VR devices is essential for creating immersive experiences in AR/VR Applications. This task is challenging due to the limited in-put from Head Mounted Devices, which capture only sparse observations from the head and hands. Predicting the full-body avatars, particularly the lower body, from these sparse observations presents significant difficulties. In this paper, we are inspired by the inherent property of the kinematic tree defined in the Skinned Multi-Person Linear (SMPL) model, where the upper body and lower body share only one common ancestor node, bringing the potential of de-coupled reconstruction. We propose a stratified approach to decouple the conventional full-body avatar reconstruction pipeline into two stages, with the reconstruction of the up-per body first and a subsequent reconstruction of the lower body conditioned on the previous stage. To implement this straightforward idea, we leverage the latent diffusion model as a powerful probabilistic generator, and train it to fol-low the latent distribution of decoupled motions explored by a VQ-VAE encoder-decoder model. Extensive experiments on AMASS mocap dataset demonstrate our state-of-the-art performance in the reconstruction of full-body motions.Keywords
All Related Versions
This publication has 35 references indexed in Scilit:
- Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model2023
- Generating Human Motion from Textual Descriptions with Discrete Representations2023
- PoseGPT: Quantization-Based 3D Human Motion Generation and Forecasting2022
- AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensing2022
- MotionCLIP: Exposing Human Motion Generation to CLIP Space2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensors2022
- FLAG: Flow-based 3D Avatar Generation from Sparse Observations2022
- Pose-conditioned joint angle limits for 3D human pose reconstruction2015
- HumanEva: Synchronized Video and Motion Capture Dataset and Baseline Algorithm for Evaluation of Articulated Human MotionInternational Journal of Computer Vision, 2009