Contextual Attention for Robust Audio-Visual Emotion Recognition
Open Access
- 26 December 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Open Journal of Signal Processing
- Vol. 7, 42-53
Abstract
Audio-visual emotion recognition (AVER) often performs well under ideal conditions but faces significant challenges in scenarios with missing modalities (e.g., missing frames of audio and/or video). Addressing these challenges is crucial for the effective deployment of AVER systems in human-computer interaction (HCI) applications, where robustness can significantly impact user experience. This study introduces a novel approach that enhances AVER robustness by leveraging a decoder-like summarizer structure. This structure processes audio and visual content and generates contextual summaries that effectively capture emotional cues even when modalities are degraded. To enhance system resilience against missing modalities, we integrate modality dropout during training, enabling the summarizer to adaptively handle these scenarios. We define the context summary length as the number of learnable query tokens used in the summarizer, a fixed hyperparameter in our model. We analyze how varying context summary lengths affect performance, identifying an optimal balance between compression and expressiveness. In addition to improving robustness, we systematically evaluate model calibration across emotions in current state-of-the-art (SOTA) AVER methods. Our experiments on the MSP-IMPROV and CREMA-D databases demonstrate that our model achieves superior performance across macro-, micro-, and weighted-F1 scores, both under ideal conditions and in scenarios with modality losses. Additionally, we conduct ablation studies to assess the impact of different context lengths on our summarizer structure in terms of overall AVER performance.Keywords
Funding Information
- National Science Foundation (CNS-2016719)
This publication has 65 references indexed in Scilit:
- Learning Trimodal Relation for Audio-Visual Question Answering with Missing Modality2024
- Semantic Space Theory: A Computational Approach to EmotionTrends in Cognitive Sciences, 2021
- Blended Emotion in-the-Wild: Multi-label Facial Expression Recognition Using Crowdsourced Annotations and Deep Locality Feature LearningInternational Journal of Computer Vision, 2018
- Formulating emotion perception as a probabilistic model with application to categorical emotion classification2017
- Self-report captures 27 distinct categories of emotion bridged by continuous gradientsProceedings of the National Academy of Sciences of the United States of America, 2017
- Missing Modalities Imputation via Cascaded Residual Autoencoder2017
- Using ensemble classifier systems for handling missing data in emotion recognition from physiology: One step towards a practical system2009
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008
- An analysis of facial expression recognition under partial facial image occlusionImage and Vision Computing, 2008
- Recognizing action units for facial expression analysisIEEE Transactions on Pattern Analysis and Machine Intelligence, 2001