Collaborative Regression of Expressive Bodies using Moderation
Authors:
- 1 December 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 792-804
Abstract
Recovering expressive humans from images is essential for understanding human behavior. Methods that estimate 3D bodies, faces, or hands have progressed significantly, yet separately. Face methods recover accurate 3D shape and geometric details, but need a tight crop and struggle with extreme views and low resolution. Whole-body methods are robust to a wide range of poses and resolutions, but provide only a rough 3D face shape without details like wrinkles. To get the best of both worlds, we introduce PIXIE, which produces animatable, whole-body 3D avatars with realistic facial detail, from a single image. For this, PIXIE uses two key observations. First, existing work combines independent estimates from body, face, and hand experts, by trusting them equally. PIXIE introduces a novel moderator that merges the features of the experts, weighted by their confidence. All part experts can contribute to the whole, using SMPL-X’s shared shape space across all body parts. Second, human shape is highly correlated with gender, but existing work ignores this. We label training images as male, female, or non-binary, and train PIXIE to infer “gendered” 3D body shapes with a novel shape loss. In addition to 3D body pose and shape parameters, PIXIE estimates expression, illumination, albedo and 3D facial surface displacements. Quantitative and qualitative evaluation shows that PIXIE estimates more accurate whole-shape and detailed face shape than the state of the art. Models and code are available at pixie.is.tue.mpg.de.Keywords
All Related Versions
This publication has 85 references indexed in Scilit:
- Maximum-Margin Structured Learning with Deep Networks for 3D Human Pose Estimation2015
- DEX: Deep EXpectation of Apparent Age from a Single Image2015
- SMPLACM Transactions on Graphics, 2015
- Pose-conditioned joint angle limits for 3D human pose reconstruction2015
- Inverse Rendering of Faces with a 3D Morphable ModelIEEE Transactions on Pattern Analysis and Machine Intelligence, 2012
- Predicting 3D People from 2D Pictures2006
- Recovering 3D human pose from monocular imagesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2005
- Estimating coloured 3D face models from single images: An example based approach1998
- Determination of 3D human body postures from a single viewComputer Vision, Graphics, and Image Processing, 1985
- Random sample consensusCommunications of the ACM, 1981