Unleashing the Transferability Power of Unsupervised Pre-Training for Emotion Recognition in Masked and Unmasked Facial Images
Open Access
- 23 August 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 11, 90876-90890
Abstract
Facial expressions are an essential part of nonverbal communication and major indicators of human emotions. Effective automatic Facial Emotion Recognition (FER) systems can facilitate comprehension of an individual’s intention, and prospective behaviors in Human-Computer and Human-Robot Interaction. However, FER faces an enduring challenge, commonly encountered in real-life, of partial occlusions caused by objects such as sunglasses and hands. With the onset of the COVID-19 pandemic, facial masks become a major obstruction for FER systems. The utilization of facial masks exacerbates the occlusion issue since these cover a significant portion of a person’s face, including the highly informative mouth area from which positive and negative emotions can be differentiated. Conversely, the efficacy of FER is largely contingent upon the supervised learning paradigm, which necessitates costly and laborious data annotation. Our study centers on utilizing the reconstruction capability of a Convolutional Residual Autoencoder to differentiate between positive and negative emotions. The proposed approach employs unsupervised feature learning and takes as inputs facial images of individuals with masks and without masks. Our study puts particular emphasis on the transferability of the proposed approach to different domains in comparison to current state-of-the-art fully supervised methods. The comprehensive experimental evaluation demonstrates the superior transferability of the proposed approach, highlighting the effectiveness of the unsupervised feature learning pipeline. Despite outperforming more complex methods in some scenarios, the proposed approach is characterized by relatively low computational expense. The source code of the proposed approach, along with the facial images created for this study, will be publicly accessible following the acceptance of this paper .Keywords
Funding Information
- European Union H2020 SPRING Project (GA #871245)
- Fondazione Valorizzazione Ricerca
- Assistive Robots wiTh EMotIonal Skills
This publication has 62 references indexed in Scilit:
- Perception of Emotions and Body Movement in the Emilya DatabaseIEEE Transactions on Affective Computing, 2016
- Adaptive Body Gesture Representation for Automatic Emotion RecognitionACM Transactions on Interactive Intelligent Systems, 2016
- Classifying imbalanced data sets using similarity based hierarchical decompositionPattern Recognition, 2015
- Automatic Nonverbal Behavior Indicators of Depression and PTSD: Exploring Gender Differences2013
- Attentional and emotional regulation in human-robot interaction2012
- Internal representations reveal cultural diversity in expectations of facial expressions of emotion.Journal of Experimental Psychology: General, 2012
- An Introduction to Restricted Boltzmann Machines2012
- Emotion recognition in human-computer interactionIEEE Signal Processing Magazine, 2001
- A circumplex model of affect.Journal of Personality and Social Psychology, 1980
- Emotion recognition: The role of facial movement and the relative importance of upper and lower areas of the face.Journal of Personality and Social Psychology, 1979