Learning To Disambiguate Strongly Interacting Hands via Probabilistic Per-Pixel Part Segmentation
Authors:
- 1 December 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-10
Abstract
In natural conversation and interaction, our hands often overlap or are in contact with each other. Due to the homogeneous appearance of hands, this makes estimating the 3D pose of interacting hands from images difficult. In this paper we demonstrate that self-similarity, and the resulting ambiguities in assigning pixel observations to the respective hands and their parts, is a major cause of the final 3D pose error. Motivated by this insight, we propose DIGIT, a novel method for estimating the 3D poses of two interacting hands from a single monocular image. The method consists of two interwoven branches that process the input imagery into a per-pixel semantic part segmentation mask and a visual feature volume. In contrast to prior work, we do not decouple the segmentation from the pose estimation stage, but rather leverage the per-pixel probabilities directly in the downstream pose estimation task. To do so, the part probabilities are merged with the visual features and processed via fully-convolutional layers. We experimentally show that the proposed approach achieves new state-of-the-art performance on the InterHand2.6M [33] dataset. We provide detailed ablation studies to demonstrate the efficacy of our method and to provide insights into how the modelling of pixel ownership affects 3D hand pose estimation.Keywords
All Related Versions
This publication has 48 references indexed in Scilit:
- Embodied handsACM Transactions on Graphics, 2017
- Hand pose estimation through semi-supervised and weakly-supervised learningComputer Vision and Image Understanding, 2017
- Learning to Estimate 3D Hand Pose from Single RGB Images2017
- Hand Keypoint Detection in Single Images Using Multiview Bootstrapping2017
- Capturing Hands in Action Using Discriminative Salient Points and Physics SimulationInternational Journal of Computer Vision, 2016
- Tracking the articulated motion of two strongly interacting hands2012
- Full DOF tracking of a hand interacting with an object by modeling occlusions and physical constraints2011
- Model-Based 3D Hand Pose Estimation from Monocular VideoIEEE Transactions on Pattern Analysis and Machine Intelligence, 2011
- Estimating 3D hand pose from a cluttered image2003
- Towards 3D hand tracking using a deformable model2002