Automatic Frustration Detection Using Thermal Imaging
Authors:
- 7 March 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 451-459
Abstract
To achieve seamless interactions, robots have to be capable of reliably detecting affective states in real time. One of the possible states that humans go through while interacting with robots is frustration. Detecting frustration from RGB images can be challenging in some real-world situations; thus, we investigate in this work whether thermal imaging can be used to create a model that is capable of detecting frustration induced by cognitive load and failure. To train our model, we collected a data set from 18 participants experiencing both types of frustration induced by a robot. The model was tested using features from several modalities: thermal, RGB, Electrodermal Activity (EDA), and all three combined. When data from both frustration cases were combined and used as training input, the model reached an accuracy of 89% with just RGB features, 87% using only thermal features, 84% using EDA, and 86% when using all modalities. Furthermore, the highest accuracy for the thermal data was reached using three facial regions of interest: nose, forehead and lower lip.Keywords
Funding Information
- Swedish Research Council (2017-05189)
- Swedish Foundation for Strategic Research (SSFFFL18-0199)
- Knut and Alice Wallenberg Foundation
This publication has 47 references indexed in Scilit:
- Training and assessing classification rules with imbalanced dataData Mining and Knowledge Discovery, 2012
- An affective computing approach to physiological emotion specificity: Toward subject‐independent and stimulus‐independent classification of film‐induced emotionsPsychophysiology, 2011
- Automatic prediction of frustrationInternational Journal of Human-Computer Studies, 2007
- Workplace user frustration with computers: an exploratory investigation of the causes and severityBehaviour & Information Technology, 2006
- Affective computing: challengesInternational Journal of Human-Computer Studies, 2003
- Toward machine emotional intelligence: analysis of affective physiological stateIEEE Transactions on Pattern Analysis and Machine Intelligence, 2001
- Using facial skin temperature to objectively evaluate sensationsInternational Journal of Industrial Ergonomics, 1997
- Autonomic nervous system response patterns specificity to basic emotionsJournal of the Autonomic Nervous System, 1997
- Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research1988
- Consistency of the factorial structures of personality ratings from different sources.The Journal of Abnormal and Social Psychology, 1949