Augmented Skeleton Space Transfer for Depth-Based Hand Pose Estimation
Authors:
- Seungryul BaekSBSeungryul Baek
- Kwang In KimKKKwang In Kim
- Tae-Kyun KimTKTae-Kyun Kim
- 1 June 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8330-8339
Abstract
Crucial to the success of training a depth-based 3D hand pose estimator (HPE) is the availability of comprehensive datasets covering diverse camera perspectives, shapes, and pose variations. However, collecting such annotated datasets is challenging. We propose to complete existing databases by generating new database entries. The key idea is to synthesize data in the skeleton space (instead of doing so in the depth-map space) which enables an easy and intuitive way of manipulating data entries. Since the skeleton entries generated in this way do not have the corresponding depth map entries, we exploit them by training a separate hand pose generator (HPG) which synthesizes the depth map from the skeleton entries. By training the HPG and HPE in a single unified optimization framework enforcing that 1) the HPE agrees with the paired depth and skeleton entries; and 2) the HPG-HPE combination satisfies the cyclic consistency (both the input and the output of HPG-HPE are skeletons) observed via the newly generated unpaired skeletons, our algorithm constructs a HPE which is robust to variations that go beyond the coverage of the existing database. Our training algorithm adopts the generative adversarial networks (GAN) training process. As a by-product, we obtain a hand pose discriminator (HPD) that is capable of picking out realistic hand poses. Our algorithm exploits this capability to refine the initial skeleton estimates in testing, further improving the accuracy. We test our algorithm on four challenging benchmark datasets (ICVL, MSRA, NYU and Big Hand 2.2M datasets) and demonstrate that our approach outperforms or is on par with state-of-the-art methods quantitatively and qualitatively.Keywords
All Related Versions
This publication has 36 references indexed in Scilit:
- Robust Articulated‐ICP for Real‐Time Hand TrackingComputer Graphics Forum, 2015
- Fast and robust hand tracking using detection-guided optimization2015
- Learning a non-linear knowledge transfer model for cross-view action recognition2015
- Learning to generate chairs with convolutional neural networks2015
- Real-Time Continuous Pose Recovery of Human Hands Using Convolutional NetworksACM Transactions on Graphics, 2014
- Real-Time Articulated Hand Pose Estimation Using Semi-supervised Transductive Regression Forests2013
- Dynamics based 3D skeletal hand tracking2013
- The Effect of Hand Dimensions, Hand Shape and Some Anthropometric Characteristics on Handgrip Strength in Male Grip Athletes and Non-AthletesJournal of Human Kinetics, 2011
- Efficient model-based 3D tracking of hand articulations using Kinect2011
- Effects of hand shape on maximal isometric grip strength and its reliability in teenagersJournal of Hand Therapy, 2005