mmHand: 3D Hand Pose Estimation Leveraging mmWave Signals
Authors:
- 23 July 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1062-1073
Abstract
Hand pose estimation is a key support for a variety of interactive applications including user interface control, sign language understanding, virtual reality modeling, etc. Existing approaches mainly exploit wearable devices such as gloves or bracelets to estimate hand poses, which may introduce high deploying costs and intrusive user experience. Others rely on vision technologies whereas they could face complicated illuminations and privacy leakage. In this paper, we present a millimeter wave (mmWave) signal-based 3D hand pose estimation system, mmHand, which utilizes a mmWave radar to generate 3D hand skeletons and reconstruct 3D hand meshes. mmHand first leverages mmWave signals to sense a hand and pre-process the signals. Then, mmHand extracts spatial and temporal features using a designed attention-based hourglass network (mmSpaceNet) and Long Short-Term Memory (LSTM), respectively. Based on the extracted features, mmHand further regresses hand joints in 3D space to generate 3D hand skeletons. Finally, 3D hand meshes that continuously describe hand poses with detailed surfaces are reconstructed through a hand Model with Articulated and Non-rigid defOrmations (MANO). Extensive experiments demonstrate that mmHand can accurately generate 3D hand skeletons with 18.3mm mean per joint position error and 95.1 % of correct key points, which indicates the effectiveness of mmHand on hand pose estimation.Keywords
Funding Information
- National Natural Science Foundation of China (62172277)
This publication has 38 references indexed in Scilit:
- MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image2022
- mPose: Environment- and subject-agnostic 3D skeleton posture reconstruction leveraging a single mmWave deviceSmart Health, 2022
- Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registration2021
- Evaluating the accuracy of hand models obtained from two 3D scanning techniquesScientific Reports, 2020
- Global Second-Order Pooling Convolutional Networks2019
- HBE: Hand Branch Ensemble Network for Real-Time 3D Hand Pose Estimation2018
- Squeeze-and-Excitation Networks2018
- Crossing Nets: Combining GANs and VAEs with a Shared Latent Space for Hand Pose Estimation2017
- Cascaded hand pose regression2015
- Latent Regression Forest: Structured Estimation of 3D Articulated Hand Posture2014