HandJoKe: Joint-Guided Keypoint Denoising Transformer for Depth-Based 3D Hand Pose Estimation
- 13 August 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 36 (1) , 861-873
Abstract
Existing depth-based 3D hand pose estimation methods typically estimate hand joints from either 2D depth images or 3D point clouds, whereas the approaches that fuse multimodal data remain underexplored. Furthermore, previous methods often struggle to learn geometric-facilitated features and precise joint correlations, especially for occluded hands, due to the lack of explicit prior guidance and insufficient cross-dimensional interaction. By taking advantage of multi-modal fusion, cross-dimensional interaction, and prior guidance, we propose a novel joint-guided keypoint denoising Transformer (named HandJoKe) to achieve more precise hand pose estimation, which can iteratively estimate hand poses based on keypoint features from both 2D depth images and 3D point clouds under explicit joint guidance within only several denoising steps. Rather than directly applying existing multi-modal fusion to perform redundant interactions among many background pixels and irrelevant points, HandJoKe focuses on modeling correlations and capturing dependencies among local informative hand regions (i.e., keypoints), thus attaining higher learning capability with lower computation redundancy. Moreover, a novel joint-guided denoising estimation strategy is introduced to adequately fuse cross-modal keypoint features under explicit joint guidance, achieving geometric-facilitated cross-modal keypoint interaction in both 2D and 3D spaces. The effectiveness of joint guidance can be further strengthened through iterative denoising, since it can subsequently update cross-modal keypoint features based on previous denoised hand poses and thus can help better locate confused joints, especially for occluded hands. Extensive experiments show that HandJoKe has achieved state-of-the-art performance on four public challenging benchmarks, including single-hand datasets NYU and ICVL, and hand-object datasets DexYCB and HO3D.Keywords
Funding Information
- National Natural Science Foundation of China (62206035)
- National Natural Science Foundation of China (62331008)
- National Natural Science Foundation of China (62476048)
- Science and Technology Innovation Key Research and Development Program of Chongqing (CSTB2024TIAD-STX0023)
- Science and Technology Innovation Key Research and Development Program of Chongqing (CSTB2023TIAD-STX0016)
- Natural Science Foundation of Chongqing Municipality (CSTB2024NSCQ-QCXMX0060)
- Natural Science Foundation of Chongqing Municipality (CSTB2023NSCQ-LZX0047)
- Natural Science Foundation of Chongqing Municipality (CSTB2022NSCQ-MSX0547)
- China Postdoctoral Science Foundation (2025MD774186)
- Chongqing Special Postdoctoral Research Funding (2024CQBSHTB2002)
- Science and Technology Research Program of Chongqing Municipal Education Commission (KJQN202200612)
This publication has 48 references indexed in Scilit:
- HandDAGT: A Denoising Adaptive Graph Transformer for 3D Hand Pose Estimation2024
- TriHorn-Net: A model for accurate depth-based 3D hand pose estimationExpert Systems with Applications, 2023
- Spatial-aware stacked regression network for real-time 3D hand pose estimationNeurocomputing, 2021
- Hand-Transformer: Non-Autoregressive Structured Modeling for 3D Hand Pose Estimation2020
- Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraints2020
- JGR-P2O: Joint Graph Reasoning Based Pixel-to-Offset Prediction Network for 3D Hand Pose Estimation from a Single Depth Image2020
- Pose guided structured region ensemble network for cascaded hand pose estimationNeurocomputing, 2020
- Point Auto-Encoder and Its Application to 2D-3D Transformation2019
- Point-to-Point Regression PointNet for 3D Hand Pose Estimation2018
- SHPR-Net: Deep Semantic Hand Pose Regression From Point CloudsIEEE Access, 2018