Typing on Any Surface: Real-Time Keystroke Detection in Augmented Reality
Authors:
- 17 January 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 350-354
Abstract
The ineffectiveness of text entry interfaces remains a significant barrier to social engagement in augmented reality (AR). Popular options, such as mid-air keyboard interface, wireless keyboards or voice input face challenges such as ergonomic issues, limited accuracy, or social discomfort in public use. This paper introduces a deep-learning method allowing AR applications to predict keystrokes from the user perspective video stream captured by any AR headset. This enables users to type on flat surfaces without a physical or virtual keyboard. Our two-stage model combines an off-the-shelf hand landmark extractor with an innovative adaptive Convolutional Recurrent Neural Network (C-RNN). It was trained on a newly built dataset, enabling prediction of 27 keys (alphabet and space) at approximately 32 FPS. With practice, users can reach a 91.0% accuracy at 40 words per minute, comparable to typing on a physical keyboard. The encouraging results demonstrate our method’s feasibility and potential for integration in diverse applications. We also explore limitations and future research directions for production system implementation.Keywords
This publication has 18 references indexed in Scilit:
- Exploration of Hands-free Text Entry Techniques For Virtual Reality2020
- MyoKey: Surface Electromyography and Inertial Motion Sensing-based Text Entry in AR2020
- Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioning2019
- RingText: Dwell-free and hands-free Text Entry for Mobile Head-Mounted Displays using Head MotionsIEEE Transactions on Visualization and Computer Graphics, 2019
- State-of-the-Art Speech Recognition with Sequence-to-Sequence Models2018
- MRTouch: Adding Touch Input to Head-Mounted Mixed RealityIEEE Transactions on Visualization and Computer Graphics, 2018
- Improving Dwell-Based Gaze Typing with Dynamic, Cascading Dwell Times2017
- SWIFTER: Design and evaluation of a speech-based text input metaphor for immersive virtual environments2016
- Vulture2014
- Selection-Based Mid-Air Text Entry on Large Displays2013