HOLD: Category-Agnostic 3D Reconstruction of Interacting Hands and Objects from Video
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 494-504
Abstract
Since humans interact with diverse objects every day, the holistic 3D capture of these interactions is important to understand and model human behaviour. However, most ex-isting methods for hand-object reconstruction from RGB ei-ther assume pre-scanned object templates or heavily rely on limited 3D hand-object data, restricting their ability to scale and generalize to more unconstrained interaction settings. To address this, we introduce HOLD the first category-agnostic method that reconstructs an articulated hand and an object jointly from a monocular interaction video. We develop a compositional articulated implicit model that can reconstruct disentangled 3D hands and ob-jects from 2D images. We also further incorporate hand-object constraints to improve hand-object poses and con-sequently the reconstruction quality. Our method does not rely on any 3D hand-object annotations while significantly outperforming fully-supervised baselines in both in-the-lab and challenging in-the-wild settings. Moreover, we qualita-tively show its robustness in reconstructing from in-the-wild videos. See here for code, data, models, and updates.Keywords
All Related Versions
This publication has 65 references indexed in Scilit:
- 3D Interacting Hand Pose Estimation by Hand De-occlusion and Removal2022
- Learning To Disambiguate Strongly Interacting Hands via Probabilistic Per-Pixel Part Segmentation2021
- Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos2021
- Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraints2020
- Category Level Object Pose Estimation via Neural Analysis-by-Synthesis2020
- InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image2020
- Hand Pose Estimation via Latent 2.5D Heatmap Regression2018
- Amodal Instance Segmentation2016
- A Comparison of Directional Distances for Hand Pose EstimationLecture Notes in Computer Science, 2013
- Visual tracking of high DOF articulated structures: An application to human hand tracking1994