Real-Time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor
Authors:
- Franziska MuellerFMFranziska Mueller
- Dushyant MehtaDMDushyant Mehta
- Oleksandr SotnychenkoOSOleksandr Sotnychenko
- Srinath SridharSSSrinath Sridhar
- Dan CasasDCDan Casas
- Christian TheobaltCTChristian Theobalt
- 1 October 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1163-1172
Abstract
We present an approach for real-time, robust and accurate hand pose estimation from moving egocentric RGB-D cameras in cluttered real environments. Existing methods typically fail for hand-object interactions in cluttered scenes imaged from egocentric viewpoints-common for virtual or augmented reality applications. Our approach uses two subsequently applied Convolutional Neural Networks (CNNs) to localize the hand and regress 3D joint locations. Hand localization is achieved by using a CNN to estimate the 2D position of the hand center in the input, even in the presence of clutter and occlusions. The localized hand position, together with the corresponding input depth value, is used to generate a normalized cropped image that is fed into a second CNN to regress relative 3D hand joint locations in real time. For added accuracy, robustness and temporal stability, we refine the pose estimates using a kinematic pose tracking energy. To train the CNNs, we introduce a new photorealistic dataset that uses a merged reality approach to capture and synthesize large amounts of annotated data of natural hand interaction in cluttered scenes. Through quantitative and qualitative evaluation, we show that our method is robust to self-occlusion and occlusions by objects, particularly in moving egocentric perspectives.Keywords
All Related Versions
This publication has 35 references indexed in Scilit:
- Robust Articulated‐ICP for Real‐Time Hand TrackingComputer Graphics Forum, 2015
- Investigating the Dexterity of Multi-Finger Input for Mid-Air Text Entry2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Real-Time Continuous Pose Recovery of Human Hands Using Convolutional NetworksACM Transactions on Graphics, 2014
- Scalable 3D Tracking of Multiple Interacting Objects2014
- Interactive Markerless Articulated Hand Motion Tracking Using RGB and Depth Data2013
- Robust realtime physics-based motion control for human graspingACM Transactions on Graphics, 2013
- Motion Capture of Hands in Action Using Discriminative Salient Points2012
- 6D hands2011
- Efficient model-based 3D tracking of hand articulations using Kinect2011