HACD: Hand-Aware Conditional Diffusion for Monocular Hand-Held Object Reconstruction
Authors:
Preprint
Abstract
Reconstructing hand-held objects from a single RGB image without known 3D object templates, category prior, or depth information is a vital yet challenging problem in computer vision. In contrast to prior works that utilize deterministic modeling paradigms, which make it hard to account for the uncertainties introduced by hand- and self-occlusion, we employ a probabilistic point cloud denoising diffusion model to tackle the above challenge. In this work, we present Hand-Aware Conditional Diffusion for monocular hand-held object reconstruction (HACD), modeling the hand-object interaction in two aspects. First, we introduce hand-aware conditioning to model hand-object interaction from both semantic and geometric perspectives. Specifically, a unified hand-object semantic embedding compensates for the 2D local feature deficiency induced by hand occlusion, and a hand articulation embedding further encodes the relationship between object vertices and hand joints. Second, we propose a hand-constrained centroid fixing scheme, which utilizes hand vertices priors to restrict the centroid deviation of partially denoised point cloud during diffusion and reverse process. Removing the centroid bias interference allows the diffusion models to focus on the reconstruction of shape, thus enhancing the stability and precision of local feature projection. Experiments on the synthetic ObMan dataset and two real-world datasets, HO3D and MOW, demonstrate our approach surpasses all existing methods by a large margin.Keywords
All Related Versions
This publication has 46 references indexed in Scilit:
- AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction2022
- ContactPose: A Dataset of Grasps with Object Contact and Hand Pose2020
- Grasping Field: Learning Implicit Representations for Human Grasps2020
- Pix2Vox++: Multi-scale Context-aware 3D Object Reconstruction from Single and Multiple ImagesInternational Journal of Computer Vision, 2020
- Learning Category-Specific Mesh Reconstruction from Image Collections2018
- Hand Pose Estimation via Latent 2.5D Heatmap Regression2018
- Real-Time Joint Tracking of a Hand Manipulating an Object from RGB-D Input2016
- Learning a Predictable and Generative Vector Representation for Objects2016
- Capturing Hands in Action Using Discriminative Salient Points and Physics SimulationInternational Journal of Computer Vision, 2016
- 3D Hand Pose Detection in Egocentric RGB-D Images2015