Facial Motion Capture System Based on Facial Electromyogram and Electrooculogram for Immersive Social Virtual Reality Applications
Open Access
Abstract
With the rapid development of virtual reality (VR) technology and the market growth of social network services (SNS), VR-based SNS have been actively developed, in which 3D avatars interact with each other on behalf of the users. To provide the users with more immersive experiences in a metaverse, facial recognition technologies that can reproduce the user’s facial gestures on their personal avatar are required. However, it is generally difficult to employ traditional camera-based facial tracking technology to recognize the facial expressions of VR users because a large portion of the user’s face is occluded by a VR head-mounted display (HMD). To address this issue, attempts have been made to recognize users’ facial expressions based on facial electromyogram (fEMG) recorded around the eyes. fEMG-based facial expression recognition (FER) technology requires only tiny electrodes that can be readily embedded in the HMD pad that is in contact with the user’s facial skin. Additionally, electrodes recording fEMG signals can simultaneously acquire electrooculogram (EOG) signals, which can be used to track the user’s eyeball movements and detect eye blinks. In this study, we implemented an fEMG- and EOG-based FER system using ten electrodes arranged around the eyes, assuming a commercial VR HMD device. Our FER system could continuously capture various facial motions, including five different lip motions and two different eyebrow motions, from fEMG signals. Unlike previous fEMG-based FER systems that simply classified discrete expressions, with the proposed FER system, natural facial expressions could be continuously projected on the 3D avatar face using machine-learning-based regression with a new concept named the virtual blend shape weight, making it unnecessary to simultaneously record fEMG and camera images for each user. An EOG-based eye tracking system was also implemented for the detection of eye blinks and eye gaze directions using the same electrodes. These two technologies were simultaneously employed to implement a real-time facial motion capture system, which could successfully replicate the user’s facial expressions on a realistic avatar face in real time. To the best of our knowledge, the concurrent use of fEMG and EOG for facial motion capture has not been reported before.Keywords
Funding Information
- Defense Acquisition Program Administration (DAPA) (KRIT-CT-21-027)
This publication has 28 references indexed in Scilit:
- Robust Facial Expression Recognition for MuCI: A Comprehensive Neuromuscular Signal AnalysisIEEE Transactions on Affective Computing, 2016
- Detection of eye blink artifacts from single prefrontal channel electroencephalogramComputer Methods and Programs in Biomedicine, 2016
- Eyebrow emotional expression recognition using surface EMG signalsNeurocomputing, 2015
- Facial Expression Recognition with sEMG Method2015
- Virtual-Reality-Based Social Interaction Training for Children with High-Functioning AutismThe Journal of Educational Research, 2013
- Classification of covariance matrices using a Riemannian-based kernel for BCI applicationsNeurocomputing, 2013
- Learning controls for blend shape based realistic facial animation2005
- EOG correction: A comparison of four methodsPsychophysiology, 2005
- The impact of avatar realism and eye gaze control on perceived quality of communication in a shared immersive virtual environment2003
- Towards an EOG-based eye tracker for computer control1998