2D Camera-Based Air-Writing Recognition Using Hand Pose Estimation and Hybrid Deep Learning Model
Open Access
- 16 February 2023
- journal article
- research article
- Published by MDPI AG in Electronics
- Vol. 12 (4) , 995
Abstract
Air-writing is a modern human–computer interaction technology that allows participants to write words or letters with finger or hand movements in free space in a simple and intuitive manner. Air-writing recognition is a particular case of gesture recognition in which gestures can be matched to write characters and digits in the air. Air-written characters show extensive variations depending on the various writing styles of participants and their speed of articulation, which presents quite a difficult task for effective character recognition. In order to solve these difficulties, this current work proposes an air-writing system using a web camera. The proposed system consists of two parts: alphabetic recognition and digit recognition. In order to assess our proposed system, two character datasets were used: an alphabetic dataset and a numeric dataset. We collected samples from 17 participants and asked each participant to write alphabetic characters (A to Z) and numeric digits (0 to 9) about 5–10 times. At the same time, we recorded the position of the fingertips using MediaPipe. As a result, we collected 3166 samples for the alphabetic dataset and 1212 samples for the digit dataset. First, we preprocessed the dataset and then created two datasets: image data and padding sequential data. The image data were fed into the convolution neural networks (CNN) model, whereas the sequential data were fed into bidirectional long short-term memory (BiLSTM). After that, we combined these two models and trained again with 5-fold cross-validation in order to increase the character recognition accuracy. In this work, this combined model is referred to as a hybrid deep learning model. Finally, the experimental results showed that our proposed system achieved an alphabet recognition accuracy of 99.3% and a digit recognition accuracy of 99.5%. We also validated our proposed system using another publicly available 6DMG dataset. Our proposed system provided better recognition accuracy compared to the existing system.Keywords
Funding Information
- MSIT (Ministry of Science and ICT), Korea (IITP-2023-2020-0-01791)
- Competitive Research Fund of The University of Aizu, Japan (IITP-2023-2020-0-01791)
This publication has 22 references indexed in Scilit:
- A Long Term Memory Recognition Framework on Multi-Complexity Motion Gestures2017
- A real-time weather forecasting and analysis2017
- Air-Writing Recognition—Part I: Modeling and Recognition of Characters, Words, and Connecting MotionsIEEE Transactions on Human-Machine Systems, 2015
- Hand Gesture and Character Recognition Based on Kinect SensorInternational Journal of Distributed Sensor Networks, 2014
- AirwritingXRDS: Crossroads, The ACM Magazine for Students, 2013
- Airwriting: Hands-Free Mobile Text Input by Spotting and Continuous Recognition of 3d-Space Handwriting with Inertial Sensors2012
- 6DMG2012
- Airwriting recognition using wearable motion sensors2010
- A letter input system based on handwriting gesturesElectronics and Communications in Japan (Part III: Fundamental Electronic Science), 2006
- Comparing the immediate usability of graffiti 2 and virtual keyboard2004