Hand-Object Pose Estimation and Reconstruction Based on Signed Distance Field and Multiscale Feature Interaction
- 29 May 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Industrial Informatics
- Vol. 20 (9) , 11242-11251
Abstract
The study of reconstruction of hands and objects from color monocular images has garnered considerable attention in recent years. In existing methods, parametric models are constructed at single scale, and the interaction between hands and objects has not fully be explored. As a result, the multiscale information in 2D images cannot be fully exploited. At the same time, the lack of feature fusion and insufficient utilization of labels also have a great impact on the reconstruction accuracy. To address the limitations, a new framework is proposed, which comprises three key modules. Firstly, a multiscale feature extractor, which generates a multiscale representation of feature, is used to capture the interaction between hand and object more effectively. Secondly, a bridge based on attention has been used to establish the connection between hand and object representations, which facilitates the integration of them. Lastly, a module based on token merge is introduced into the framework, which provides the segmentation representation of object. The experimental results on two datasets, named Obman and DexYCB, demonstrated that the proposed method had good performance and achieved a shape error about 0.121 $\text{cm}^{2}$ on Obman and 0.40 $\text{cm}^{2}$ on DexYCB, outperforming the state-of-the-art methods. This study will probably provide the human-computer interaction methods with broader application prospects.Keywords
Funding Information
- Ningxia Hui Autonomous Region Key R&D Plan (2023BEG02035, 2023BEG03043)
- Fudan University Medical Engineering Cross Research (yg2022-33)
This publication has 31 references indexed in Scilit:
- SPViT: Enabling Faster Vision Transformers via Latency-Aware Soft Token Pruning2022
- AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction2022
- A Skeleton-Driven Neural Occupancy Representation for Articulated Hands2021
- Grasping Field: Learning Implicit Representations for Human Grasps2020
- Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction2020
- Learning Joint Reconstruction of Hands and Manipulated Objects2019
- 3D Hand Shape and Pose From Images in the Wild2019
- SHPR-Net: Deep Semantic Hand Pose Regression From Point CloudsIEEE Access, 2018
- A Papier-Mache Approach to Learning 3D Surface Generation2018
- 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction2016