Multi-Mode Hand Gesture-Based VR Locomotion Technique for Intuitive Telemanipulation Viewpoint Control in Tightly Arranged Logistic Environments

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Telemanipulation-based object-side picking with a suction gripper often faces challenges such as occlusion of the target object or the gripper and the need for precise alignment between the suction cup and the object’s surface. These issues can significantly affect task success rates in logistics environments. To address these problems, this study proposes a multi-mode hand gesture-based virtual reality (VR) locomotion method to enable intuitive and precise viewpoint control. The system utilizes a head-mounted display (HMD) camera to capture hand skeleton data, which a multi-layer perceptron (MLP) model processes. The model classifies gestures into three modes: translation, rotation, and fixed, corresponding to fist, pointing, and unknown gestures, respectively. Translation mode moves the viewpoint based on the wrist’s displacement, rotation mode adjusts the viewpoint’s angle based on the wrist’s angular displacement, and fixed mode stabilizes the viewpoint when gestures are ambiguous. A dataset of 4312 frames was used for training and validation, with 666 frames for testing. The MLP model achieved a classification accuracy of 98.4%, with precision, recall, and F1-score exceeding 0.98. These results demonstrate the system’s ability to address the challenges of telemanipulation tasks by enabling accurate gesture recognition and seamless mode transitions.
Funding Information
  • National Research Foundation of Korea (NRF) funded by the Ministry of Education (RS-2022-NR073933, 2022-0-00218)
  • Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT) (RS-2022-NR073933, 2022-0-00218)

This publication has 28 references indexed in Scilit: