Deep-Learning-Derived Facial Electromyogram Signatures of Emotion in Immersive Virtual Reality (bWell): Exploring the Impact of Emotional, Cognitive, and Physical Demands
Open Access
Abstract
Emotional and workload-related states unfold dynamically during immersive virtual reality (VR) experiences, yet reliable physiological modeling in such environments remains challenging. We investigated whether multi-channel facial electromyography (fEMG), combined with spatio-temporal deep learning, can (i) accurately classify calibrated facial expressions across participants and (ii) transfer to spontaneous, task-elicited behavior in immersive VR. Twelve adults completed a calibration phase involving four intentional expressions (smile, frown, raised eyebrow, neutral), followed by VR scenes designed to elicit emotional, cognitive, physical, and dual task demands. After participant-level physiological normalization, a single shared Convolutional Neural Network–Temporal Convolutional Network (CNN–TCN) model was trained and evaluated using leave-one-participant-out (LOPO) validation. The model achieved strong cross-participant performance (Macro-F1 = 0.88 ± 0.13; ROC-AUC = 0.95 ± 0.06). When applied to unlabeled spontaneous VR task-elicited fEMG recordings, the trained model generated continuous expression classes. Derived static and temporal expression features showed scene-dependent modulation and False Discovery Rate (FDR)-surviving associations, primarily with perceived physical demand (NASA-TLX). The observed muscle activation patterns were physiologically plausible and aligned with Facial Action Coding System (FACS)-based interpretations of underlying muscle activity. These findings demonstrate that end-to-end spatio-temporal modeling of raw fEMG enables facial expression sensing in immersive VR using a single shared model following physiological normalization. The proposed framework bridges calibrated expression learning and spontaneous task-elicited behavior, supporting privacy-preserving, continuous and physiologically grounded monitoring in human-centered VR applications.Keywords
Funding Information
- Defence Research and Development Canada
This publication has 70 references indexed in Scilit:
- Exploring working memory across aging using virtual realityVirtual Reality, 2025
- Facial electromyogram-based emotion recognition for virtual reality applications using machine learning classifiers trained on posed expressionsBiomedical Engineering Letters, 2025
- Facial surface electromyography: A systematic review on the state of the art and current perspectivesAmerican Journal of Otolaryngology, 2023
- Facial electromyogram-based facial gesture recognition for hands-free control of an AR/VR environment: optimal gesture set selection and validation of feasibility as an assistive technologyBiomedical Engineering Letters, 2023
- Emotion recognition using facial expressions in an immersive virtual reality applicationVirtual Reality, 2022
- The limitations for expression recognition in computer vision introduced by facial masksMultimedia Tools and Applications, 2022
- Performance enhancement of facial electromyogram-based facial-expression recognition for social virtual reality applications using linear discriminant analysis adaptationVirtual Reality, 2021
- A survey of machine learning techniques in physiology based mental stress detection systemsBiocybernetics and Biomedical Engineering, 2019
- Image-based emotion recognition using evolutionary algorithmsBiologically Inspired Cognitive Architectures, 2018
- Measuring emotion: The self-assessment manikin and the semantic differentialJournal of Behavior Therapy and Experimental Psychiatry, 1994