Online Multimodal Inference of Mental Workload for Cognitive Human Machine Systems
Open Access
Abstract
With increasingly higher levels of automation in aerospace decision support systems, it is imperative that the human operator maintains the required level of situational awareness in different operational conditions and a central role in the decision-making process. While current aerospace systems and interfaces are limited in their adaptability, a Cognitive Human Machine System (CHMS) aims to perform dynamic, real-time system adaptation by estimating the cognitive states of the human operator. Nevertheless, to reliably drive system adaptation of current and emerging aerospace systems, there is a need to accurately and repeatably estimate cognitive states, particularly for Mental Workload (MWL), in real-time. As part of this study, two sessions were performed during a Multi-Attribute Task Battery (MATB) scenario, including a session for offline calibration and validation and a session for online validation of eleven multimodal inference models of MWL. The multimodal inference model implemented included an Adaptive Neuro Fuzzy Inference System (ANFIS), which was used in different configurations to fuse data from an Electroencephalogram (EEG) model’s output, four eye activity features and a control input feature. The online validation of the ANFIS models produced good results, while the best performing model (containing all four eye activity features and the control input feature) showed an average Mean Absolute Error (MAE) = 0.67 ± 0.18 and Correlation Coefficient (CC) = 0.71 ± 0.15. The remaining six ANFIS models included data from the EEG model’s output, which had an offset discrepancy. This resulted in an equivalent offset for the online multimodal fusion. Nonetheless, the efficacy of these ANFIS models could be confirmed by the pairwise correlation with the task level, where one model demonstrated a CC = 0.77 ± 0.06, which was the highest among all of the ANFIS models tested. Hence, this study demonstrates the suitability for online multimodal fusion of features extracted from EEG signals, eye activity and control inputs to produce an accurate and repeatable inference of MWL.Keywords
Funding Information
- Thales Group (RE-03975)
- Northrop Grumman (RE-03671)
This publication has 60 references indexed in Scilit:
- Filter Bank Common Spatial Pattern Algorithm on BCI Competition IV Datasets 2a and 2bFrontiers in Neuroscience, 2012
- Driver workload and eye blink durationTransportation Research Part F: Traffic Psychology and Behaviour, 2011
- A neuroergonomic approach to evaluating mental workload in hypermedia interactionsInternational Journal of Industrial Ergonomics, 2011
- Improvement of EEG Signal Acquisition: An Electrical Aspect for State of the Art of Front EndComputational Intelligence and Neuroscience, 2010
- Situation Awareness, Mental Workload, and Trust in Automation: Viable, Empirically Supported Cognitive Engineering ConstructsJournal of Cognitive Engineering and Decision Making, 2008
- Using eye movement activity as a correlate of cognitive workloadInternational Journal of Industrial Ergonomics, 2006
- Real-Time Assessment of Mental Workload Using Psychophysiological Measures and Artificial Neural NetworksHuman Factors: The Journal of the Human Factors and Ergonomics Society, 2003
- Flight Deck Automation issuesThe International Journal of Aviation Psychology, 1999
- ANFIS: adaptive-network-based fuzzy inference systemIEEE Transactions on Systems, Man, and Cybernetics, 1993
- Measurement and analysis methods of heart rate and respiration for use in applied environmentsBiological Psychology, 1992