PoseBERT: A Generic Transformer Module for Temporal 3D Human Modeling
- 12 December 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. PP (01628828) , 1-17
Abstract
Training state-of-the-art models for human pose estimation in videos requires datasets with annotations that are really hard and expensive to obtain. Although transformers have been recently utilized for body pose sequence modeling, related methods rely on pseudo-ground truth to augment the currently limited training data available for learning such models. In this paper, we introduce PoseBERT, a transformer module that is fully trained on 3D Motion Capture (MoCap) data via masked modeling. It is simple, generic and versatile, as it can be plugged on top of any image-based model to transform it in a video-based model leveraging temporal information. We showcase variants of PoseBERT with different inputs varying from 3D skeleton keypoints to rotations of a 3D parametric model for either the full body (SMPL) or just the hands (MANO). Since PoseBERT training is task agnostic, the model can be applied to several tasks such as pose refinement, future pose prediction or motion completion without finetuning . Our experimental results validate that adding PoseBERT on top of various state-of-the-art pose estimation methods consistently improves their performances, while its low computational cost allows us to use it in a real-time demo for smoothly animating a robotic hand via a webcam. Test code and models are available at https://github.com/naver/posebert .Keywords
All Related Versions
This publication has 70 references indexed in Scilit:
- Image-Based Synthesis for Deep 3D Human Pose EstimationInternational Journal of Computer Vision, 2018
- Embodied handsACM Transactions on Graphics, 2017
- Monocular 3D Human Pose Estimation in the Wild Using Improved CNN Supervision2017
- Learning from Synthetic Humans2017
- Unite the People: Closing the Loop Between 3D and 2D Human Representations2017
- Keep It SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image2016
- Fast R-CNN2015
- SMPLACM Transactions on Graphics, 2015
- Pose-conditioned joint angle limits for 3D human pose reconstruction2015
- What Is a Savitzky-Golay Filter? [Lecture Notes]IEEE Signal Processing Magazine, 2011