GRIP: Generating Interaction Poses Using Spatial Cues and Latent Consistency
Open Access
- 18 March 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 933-943
Abstract
Hands are dexterous and highly versatile manipulators that are central to how humans interact with objects and their environment. Consequently, modeling realistic hand-object interactions, including the subtle motion of individual fingers, is critical for applications in computer graphics, computer vision, and mixed reality. Prior work on capturing and modeling humans interacting with objects in 3D focuses on the body and object motion, often ignoring hand pose. In contrast, we introduce GRIP, a learning-based method that takes, as input, the 3D motion of the body and the object, and synthesizes realistic motion for both hands before, during, and after object interaction. As a preliminary step before synthesizing the hand motion, we first use a network, ANet, to denoise the arm motion. Then, we leverage the spatio-temporal relationship between the body and the object to extract novel temporal interaction cues, and use them in a two-stage inference pipeline to generate the hand motion. In the first stage, we introduce a new approach to encourage motion temporal consistency in the latent space (LTC) and generate consistent interaction motions. In the second stage, GRIP generates refined hand poses to avoid hand-object penetrations. Given sequences of noisy body and object motion, GRIP “upgrades” them to include hand-object interaction. Quantitative experiments and perceptual studies demonstrate that GRIP outperforms baseline methods and generalizes to unseen objects and motions from different motion-capture datasets. Our models and code are available for research purposes at https://grip.is.tue.mpg.de.Keywords
All Related Versions
Funding Information
- Ministry of Education
This publication has 60 references indexed in Scilit:
- POCO: 3D Pose and Shape Estimation with Confidence2024
- TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinement2022
- InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interaction2022
- A Skeleton-Driven Neural Occupancy Representation for Articulated Hands2021
- Intention-based Long-Term Human Motion Anticipation2021
- ContactPose: A Dataset of Grasps with Object Contact and Hand Pose2020
- Grasping Field: Learning Implicit Representations for Human Grasps2020
- GRAB: A Dataset of Whole-Body Human Grasping of Objects2020
- Capturing Hands in Action Using Discriminative Salient Points and Physics SimulationInternational Journal of Computer Vision, 2016
- OpenGRASP: A Toolkit for Robot Grasping Simulation2010