PoseFix: Correcting 3D Human Poses with Natural Language
Authors:
- 1 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 14972-14982
Abstract
Automatically producing instructions to modify one’s posture could open the door to endless applications, such as personalized coaching and in-home physical therapy. Tackling the reverse problem (i.e., refining a 3D pose based on some natural language feedback) could help for assisted 3D character animation or robot teaching, for instance. Although a few recent works explore the connections between natural language and 3D human pose, none focus on describing 3D body pose differences. In this paper, we tackle the problem of correcting 3D human poses with natural language. To this end, we introduce the Pose- Fix dataset, which consists of several thousand paired 3D poses and their corresponding text feedback, that describe how the source pose needs to be modified to obtain the target pose. We demonstrate the potential of this dataset on two tasks: (1) text-based pose editing, that aims at generating corrected 3D body poses given a query pose and a text modifier; and (2) correctional text generation, where instructions are generated based on the differences between two body poses. The dataset and the code are available at https://europe.naverlabs.com/research/computer-vision/posefix/.Keywords
All Related Versions
This publication has 48 references indexed in Scilit:
- 3D Pose Based Feedback for Physical Exercises2023
- PoseScript: 3D Human Poses from Natural Language2022
- CLIP-Actor: Text-Driven Recommendation and Stylization for Animating Human Meshes2022
- TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts2022
- MotionCLIP: Exposing Human Motion Generation to CLIP Space2022
- TEMOS: Generating Diverse Human Motions from Textual Descriptions2022
- StyleGAN-Human: A Data-Centric Odyssey of Human Generation2022
- DLow: Diversifying Latent Flows for Diverse Human Motion Prediction2020
- Attributes as Operators: Factorizing Unseen Attribute-Object Compositions2018
- Microsoft COCO: Common Objects in Context2014