UniHPR: Unified Human Pose Representation via Singular Value Contrastive Learning
Preprint
Abstract
In recent years, there has been a growing interest in developing effective alignment pipelines to generate unified representations from different modalities for multi-modal fusion and generation. As an important component of Human-Centric applications, Human Pose representations are critical in many downstream tasks, such as Human Pose Estimation, Action Recognition, Human-Computer Interaction, Object tracking, etc. Human Pose representations or embeddings can be extracted from images, 2D keypoints, 3D skeletons, mesh models, and lots of other modalities. Yet, there are limited instances where the correlation among all of those representations has been clearly researched using a contrastive paradigm. In this paper, we propose UniHPR, a unified Human Pose Representation learning pipeline, which aligns Human Pose embeddings from images, 2D and 3D human poses. To align more than two data representations at the same time, we propose a novel singular value-based contrastive learning loss, which better aligns different modalities and further boosts performance. To evaluate the effectiveness of the aligned representation, we choose 2D and 3D Human Pose Estimation (HPE) as our evaluation tasks. In our evaluation, with a simple 3D human pose decoder, UniHPR achieves remarkable performance metrics: MPJPE 49.9mm on the Human3.6M dataset and PA-MPJPE 51.6mm on the 3DPW dataset with cross-domain evaluation. Meanwhile, we are able to achieve 2D and 3D pose retrieval with our unified human pose representations in Human3.6M dataset, where the retrieval error is 9.24mm in MPJPE.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- MPM: A Unified 2D-3D Human Pose Representation via Masked Pose Modeling2024
- PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation2021
- VIBE: Video Inference for Human Body Pose and Shape Estimation2020
- Exploiting Temporal Context for 3D Human Pose Estimation in the Wild2019
- Learning 3D Human Dynamics From Video2019
- 3D Human Pose Estimation in Video With Temporal Convolutions and Semi-Supervised Training2019
- Semantic Graph Convolutional Networks for 3D Human Pose Regression2019
- Learning to Estimate 3D Human Pose and Shape from a Single Color Image2018
- Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose2017
- Keep It SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image2016