An Efficient RGB-D Hand Gesture Detection Framework for Dexterous Robot Hand-Arm Teleoperation System
- 26 September 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Human-Machine Systems
- Vol. 53 (1) , 13-23
Abstract
Aiming at the problems of accurate and fast hand gesture detection and teleoperation mapping in the hand-based visual teleoperation of dexterous robots, an efficient hand gesture detection framework based on deep learning is proposed in this article. It can achieve an accurate and fast hand gesture detection and teleoperation of dexterous robots based on an anchor-free network architecture by using an RGB-D camera. First, an RGB-D early-fusion method based on the HSV space is proposed, effectively reducing background interference and enhancing hand information. Second, a hand gesture classification network (HandClasNet) is proposed to realize hand detection and localization by detecting the center and corner points of hands, and a HandClasNet is proposed to realize gesture recognition by using a parallel EfficientNet structure. Then, a dexterous robot hand-arm teleoperation system based on the hand gesture detection framework is designed to realize the hand-based teleoperation of a dexterous robot. Our method achieves high accuracy with fast speed on public and custom hand datasets and outperforms some state-of-the-art methods. In addition, the application of the proposed method in the hand-based teleoperation system can control the grasping of various objects by a dexterous hand-arm system in real time and accurately, which verifies the efficiency of our method.Keywords
Funding Information
- National Natural Science Foundation of China (62006204, 52075530)
- Basic and Applied Basic Research Foundation of Guangdong Province (2022A1515011431)
- Shenzhen Science and Technology Program (RCBS20210609104516043, JCYJ20180508162406177, JCYJ20210324115604012, JSGG20220606140201003, JSGG20210802154004014)
- Shenzhen Institute of Artificial Intelligence and Robotics for Society (AC01202101102)
- European Regional Development Fund
This publication has 30 references indexed in Scilit:
- Online optical marker-based hand tracking with deep labelsACM Transactions on Graphics, 2018
- RGB-D-based human motion recognition with deep learning: A surveyComputer Vision and Image Understanding, 2018
- Finger spelling recognition using depth information and support vector machineMultimedia Tools and Applications, 2018
- Egocentric Hand Detection Via Dynamic Region GrowingACM Transactions on Multimedia Computing, Communications, and Applications, 2017
- Multimodal Gesture Recognition Based on the ResC3D Network2017
- Focal Loss for Dense Object Detection2017
- Static Hand Gesture Recognition with Parallel CNNs for Space Human-Robot Interaction2017
- Toward an Enhanced Human–Machine Interface for Upper-Limb Prosthesis Control With Combined EMG and NIRS SignalsIEEE Transactions on Human-Machine Systems, 2017
- Glove-Based Continuous Arabic Sign Language Recognition in User-Dependent ModeIEEE Transactions on Human-Machine Systems, 2015
- Hand detection using multiple proposals2011