Jointly Learning From Unimodal and Multimodal-Rated Labels in Audio-Visual Emotion Recognition
Open Access
- 15 January 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Open Journal of Signal Processing
- Vol. 6, 165-174
Abstract
Audio-visual emotion recognition (AVER) has been an important research area in human-computer interaction (HCI). Traditionally, audio-visual emotional datasets and corresponding models derive their ground truths from annotations obtained by raters after watching the audio-visual stimuli. This conventional method, however, neglects the nuanced human perception of emotional states, which varies when annotations are made under different emotional stimuli conditions-whether through unimodal or multimodal stimuli. This study investigates the potential for enhanced AVER system performance by integrating diverse levels of annotation stimuli, reflective of varying perceptual evaluations. We propose a two-stage training method to train models with the labels elicited by audio-only, face-only, and audio-visual stimuli. Our approach utilizes different levels of annotation stimuli according to which modality is present within different layers of the model, effectively modeling annotation at the unimodal and multi-modal levels to capture the full scope of emotion perception across unimodal and multimodal contexts. We conduct the experiments and evaluate the models on the CREMA-D emotion database. The proposed methods achieved the best performances in macro-/weighted-F1 scores. Additionally, we measure the model calibration, performance bias, and fairness metrics considering the age, gender, and race of the AVER systems.Keywords
Funding Information
- National Science Foundation (CNS-2016719)
This publication has 42 references indexed in Scilit:
- Leveraging recent advances in deep learning for audio-Visual emotion recognitionPattern Recognition Letters, 2021
- Semantic Space Theory: A Computational Approach to EmotionTrends in Cognitive Sciences, 2021
- Multimodal Transformer Fusion for Continuous Emotion Recognition2020
- Class-Balanced Loss Based on Effective Number of Samples2019
- Every Rating Matters: Joint Learning of Subjective Labels and Individual Annotators for Speech Emotion Classification2019
- MobileNetV2: Inverted Residuals and Linear Bottlenecks2018
- Formulating emotion perception as a probabilistic model with application to categorical emotion classification2017
- Combining feature-level and decision-level fusion in a hierarchical classifier for emotion recognition in the wildJournal on Multimodal User Interfaces, 2015
- Emotion in the voice influences the way we scan emotional facesSpeech Communication, 2014
- Is there an advantage for recognizing multi-modal emotional stimuli?Motivation and Emotion, 2011