ARTS: Semi-Analytical Regressor using Disentangled Skeletal Representations for Human Mesh Recovery from Videos
Open Access
- 28 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1514-1523
Abstract
Although existing video-based 3D human mesh recovery methods have made significant progress, simultaneously estimating human pose and shape from low-resolution image features limits their performance. These image features lack sufficient spatial information about the human body and contain various noises (e.g., background, lighting, and clothing), which often results in inaccurate pose and inconsistent motion. Inspired by the rapid advance in human pose estimation, we discover that compared to image features, skeletons inherently contain accurate human pose and motion. Therefore, we propose a novel semi-Analytical Regressor using disenTangled Skeletal representations for human mesh recovery from videos, called ARTS. Specifically, a skeleton estimation and disentanglement module is proposed to estimate the 3D skeletons from a video and decouple them into disentangled skeletal representations (i.e., joint position, bone length, and human motion). Then, to fully utilize these representations, we introduce a semi-analytical regressor to estimate the parameters of the human mesh model. The regressor consists of three modules: Temporal Inverse Kinematics (TIK), Bone-guided Shape Fitting (BSF), and Motion-Centric Refinement (MCR). TIK utilizes joint position to estimate initial pose parameters and BSF leverages bone length to regress bone-aligned shape parameters. Finally, MCR combines human motion representation with image features to refine the initial human model parameters. Extensive experiments demonstrate that our ARTS surpasses existing state-of-the-art video-based methods in both per-frame accuracy and temporal consistency on popular benchmarks: 3DPW, MPI-INF-3DHP, and Human3.6M. Code is available at https://github.com/TangTao-PKU/ARTS>Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (No.62373009)
This publication has 44 references indexed in Scilit:
- CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation2022
- Hierarchical Kinematic Human Mesh Recovery2020
- Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose2020
- Recovering Accurate 3D Human Pose in the Wild Using IMUs and a Moving Camera2018
- Neural Body Fitting: Unifying Deep Learning and Model Based Human Pose and Shape Estimation2018
- Cascaded Pyramid Network for Multi-person Pose Estimation2018
- Learning to Estimate 3D Human Pose and Shape from a Single Color Image2018
- Stacked Hourglass Networks for Human Pose Estimation2016
- Deep Residual Learning for Image Recognition2016
- 2D Human Pose Estimation: New Benchmark and State of the Art Analysis2014