Expressive Whole-Body 3D Gaussian Avatar
Preprint
Abstract
Facial expression and hand motions are necessary to express our emotions and interact with the world. Nevertheless, most of the 3D human avatars modeled from a casually captured video only support body motions without facial expressions and hand motions.In this work, we present ExAvatar, an expressive whole-body 3D human avatar learned from a short monocular video. We design ExAvatar as a combination of the whole-body parametric mesh model (SMPL-X) and 3D Gaussian Splatting (3DGS). The main challenges are 1) a limited diversity of facial expressions and poses in the video and 2) the absence of 3D observations, such as 3D scans and RGBD images. The limited diversity in the video makes animations with novel facial expressions and poses non-trivial. In addition, the absence of 3D observations could cause significant ambiguity in human parts that are not observed in the video, which can result in noticeable artifacts under novel motions. To address them, we introduce our hybrid representation of the mesh and 3D Gaussians. Our hybrid representation treats each 3D Gaussian as a vertex on the surface with pre-defined connectivity information (i.e., triangle faces) between them following the mesh topology of SMPL-X. It makes our ExAvatar animatable with novel facial expressions by driven by the facial expression space of SMPL-X. In addition, by using connectivity-based regularizers, we significantly reduce artifacts in novel facial expressions and poses.Keywords
All Related Versions
This publication has 38 references indexed in Scilit:
- NeuMan: Neural Human Radiance Field from a Single Video2022
- MonoNHR: Monocular Neural Human Renderer2022
- Collaborative Regression of Expressive Bodies using Moderation2021
- Monocular Expressive Body Regression Through Body-Driven Attention2020
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis2020
- DeepHandMesh: A Weakly-Supervised Deep Encoder-Decoder Framework for High-Fidelity Hand Mesh Modeling2020
- DensePose: Dense Human Pose Estimation in the Wild2018
- Video Based Reconstruction of 3D People Models2018
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies2018