Monocular Interacting-Hand Reconstruction with Multimodal Context Fusion and Spatial Attention
Open Access
- 11 November 2026
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 22 (11) , 1-23
Abstract
3D hand reconstruction from monocular RGB images has attracted increasing attention due to its low cost and ease of deployment. However, accurately reconstructing interacting hands remains challenging, primarily because mutual occlusion leads to missing visual evidence, while image cropping weakens global spatial consistency. To address these issues, we propose the Context-Aware Interacting Hand Reconstruction Network (CANet), a monocular interacting-hand reconstruction framework that integrates multimodal context fusion with structured spatial attention. Specifically, CANet leverages estimated depth maps, edge contours, and center heatmaps to retain global contextual cues and guide reconstruction in occluded regions. A hierarchical spatial attention module further enhances spatial consistency by separately modeling intra-hand structural dependencies and inter-hand interactions, enabling more coherent reasoning about complex hand poses. Experiments across multiple public benchmarks demonstrate that CANet consistently improves the accuracy of both hand pose estimation and mesh reconstruction.Keywords
Funding Information
- National Natural Science Foundation of China (62576202)
- Natural Science Foundation of Shanghai (25ZR1401115)
- National Natural Science Foundation of China (82430060)
- National Key Research and Development Program of China (2024YFC3306703, 2024YFC3306700)
- Shanghai Key Laboratory of Forensic Medicine, Academy of Forensic Science, Ministry of Justice (KF202413)
- Shanghai Technical Service Center of Science and Engineering Computing, Shanghai University
This publication has 49 references indexed in Scilit:
- 3D Interacting Hand Pose Estimation by Hand De-occlusion and Removal2022
- Grasping Field: Learning Implicit Representations for Human Grasps2020
- On the Continuity of Rotation Representations in Neural Networks2019
- Learning Joint Reconstruction of Hands and Manipulated Objects2019
- Pushing the Envelope for RGB-Based Dense 3D Hand Pose Estimation via Neural Rendering2019
- 3D Hand Shape and Pose From Images in the Wild2019
- Weakly-Supervised 3D Hand Pose Estimation from Monocular RGB Images2018
- Cross-Modal Deep Variational Hand Pose Estimation2018
- GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB2018
- Capturing Hands in Action Using Discriminative Salient Points and Physics SimulationInternational Journal of Computer Vision, 2016