“I Can See Your Password”: A Case Study About Cybersecurity Risks in Mid-Air Interactions of Mixed Reality-Based Smart Manufacturing Applications
- 10 October 2023
- journal article
- research article
- Published by Wiley in Journal of Computing and Information Science in Engineering
- Vol. 24 (3) , 1-12
Abstract
This paper aims to present a potential cybersecurity risk existing in mixed reality (MR)-based smart manufacturing applications that decipher digital passwords through a single RGB camera to capture the user’s mid-air gestures. We first created a test bed, which is an MR-based smart factory management system consisting of mid-air gesture-based user interfaces (UIs) on a video see-through MR head-mounted display. To interact with UIs and input information, the user’s hand movements and gestures are tracked by the MR system. We setup the experiment to be the estimation of the password input by users through mid-air hand gestures on a virtual numeric keypad. To achieve this goal, we developed a lightweight machine learning-based hand position tracking and gesture recognition method. This method takes either video streaming or recorded video clips (taken by a single RGB camera in front of the user) as input, where the videos record the users’ hand movements and gestures but not the virtual UIs. With the assumption of the known size, position, and layout of the keypad, the machine learning method estimates the password through hand gesture recognition and finger position detection. The evaluation result indicates the effectiveness of the proposed method, with a high accuracy of 97.03%, 94.06%, and 83.83% for 2-digit, 4-digit, and 6-digit passwords, respectively, using real-time video streaming as input with known length condition. Under the unknown length condition, the proposed method reaches 85.50%, 76.15%, and 77.89% accuracy for 2-digit, 4-digit, and 6-digit passwords, respectively.Keywords
Funding Information
- Division of Graduate Education (2125362)
- Division of Information and Intelligent Systems (2222853)
- Facebook (2022 Meta Research Awards for Privacy- Enhancing Technologies.)
This publication has 33 references indexed in Scilit:
- Secure data processing for IoT middleware systemsThe Journal of Supercomputing, 2018
- Protecting Nonverbal Data Tracked in Virtual RealityJAMA Pediatrics, 2018
- Internet of Things (IoT) Cybersecurity Research: A Review of Current Research TopicsIEEE Internet of Things Journal, 2018
- A systematic review of augmented reality applications in maintenanceRobotics and Computer-Integrated Manufacturing, 2018
- Anomaly behavior analysis for IoT sensorsTransactions on Emerging Telecommunications Technologies, 2017
- Towards a griddable distributed manufacturing system with augmented reality interfacesRobotics and Computer-Integrated Manufacturing, 2016
- The use of virtual reality and physical tools in the development and validation of ease of entry and exit in passenger vehiclesApplied Ergonomics, 2015
- Design of a virtual reality training system for human–robot collaboration in manufacturing tasksInternational Journal on Interactive Design and Manufacturing (IJIDeM), 2015
- Vision-Based Hand Interaction in Augmented Reality EnvironmentInternational Journal of Human–Computer Interaction, 2011
- Long Short-Term MemoryNeural Computation, 1997