Cognitive Load Monitoring With Wearables–Lessons Learned From a Machine Learning Challenge
Open Access
- 28 June 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 9, 103325-103336
Abstract
To further extend the applicability of wearable sensors, methods for accurately extracting subtle psychological information from the sensor data are required. However, accessing subjective information in everyday life, such as cognitive load, remains challenging. To bring consensus on methods for cognitive load monitoring, a machine learning challenge is organized. The participants developed machine learning methods for cognitive load classification using wrist-worn physiological sensors' data, namely heart rate, R-R intervals, skin conductance, and skin temperature. The data from subjects solving cognitive tasks of varying difficulty is used for the challenge. This article presents a systematic comparison and multi-strategic performance evaluation of the thirteen methods submitted to this challenge. A systematic comparison of preprocessing techniques, classification algorithms, and implementation techniques is presented. Performance variations for different task difficulty levels, different subjects, and different experiment periods are evaluated. The results indicate that the most robust methods used multimodal sensor data, classical classification approaches such as decision trees and support vector machines or their ensembles, and Bayesian hyperparameter optimization for hyperparameter tuning. The most accurate models used handcrafted features that are further selected using sequential backward floating search and evaluated using stratified person-aware cross-validation strategy. Moreover, the results indicated better classification performance for specific test subjects, the tasks with the highest difficulty, and in some cases, the time elapsed since the start of the experiment. This dependency is likely due to model overfitting or due to the subjective nature of the psychophysiological process. The intersubject variability in responses is challenging to be captured through objective binary labels for cognitive load, thereby warranting more sophisticated annotation approaches.Keywords
Funding Information
- Slovenian Research Agency (U2-AG-16/0672, 0287, P2-0098, N2-0136)
- Jožef Stefan Institute and Fraunhofer IIS through the German Academic Exchange Service and ARRS, in 2019 (57451027)
- Wide Health project-Horizon 2020 (95227)
This publication has 45 references indexed in Scilit:
- COGCAM2016
- Classifying Cognitive Load and Driving Situation with Machine LearningInternational Journal of Machine Learning and Computing, 2014
- Using psycho-physiological measures to assess task difficulty in software development2014
- GSR and Blink Features for Cognitive Load Classification2013
- A comparison of four methods for cognitive load measurement2011
- Automatic cognitive load detection from speech features2007
- Why It’s Worth the Hassle: The Value of In-Situ Studies When Designing Ubicomp2007
- Physical, mental, emotional, and subjective workload components in train driversErgonomics, 1994
- The Computer for the 21st CenturyScientific American, 1991
- Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research1988