Versatile Audio-Visual Learning for Emotion Recognition
Open Access
- 25 July 2024
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Affective Computing
- Vol. 16 (1) , 306-318
Abstract
Most current audio-visual emotion recognition models lack the flexibility needed for deployment in practical applications. We envision a multimodal system that works even when only one modality is available and can be implemented interchangeably for either predicting emotional attributes or recognizing categorical emotions. Achieving such flexibility in a multimodal emotion recognition system is difficult due to the inherent challenges in accurately interpreting and integrating varied data sources. It is also a challenge to robustly handle missing or partial information while allowing direct switch between regression or classification tasks. This study proposes a versatile audio-visual learning (VAVL) framework for handling unimodal and multimodal systems for emotion regression or emotion classification tasks. We implement an audio-visual framework that can be trained even when audio and visual paired data is not available for part of the training set (i.e., audio only or only video is present). We achieve this effective representation learning with audio-visual shared layers, residual connections over shared layers, and a unimodal reconstruction task. Our experimental results reveal that our architecture significantly outperforms strong baselines on the CREMA-D, MSP-IMPROV, and CMU-MOSEI corpora. Notably, VAVL attains a new state-of-the-art performance in the emotional attribute prediction task on the MSP-IMPROV corpus.Keywords
All Related Versions
Funding Information
- NSF (CNS-2016719)
This publication has 59 references indexed in Scilit:
- Leveraging recent advances in deep learning for audio-Visual emotion recognitionPattern Recognition Letters, 2021
- Deep Audio-visual Learning: A SurveyInternational Journal of Automation and Computing, 2021
- Visual-audio emotion recognition based on multi-task and ensemble learning with multiple featuresNeurocomputing, 2020
- Audio-Visual Event Localization in Unconstrained Videos2018
- Multi-modal emotion analysis from facial expressions and electroencephalogramComputer Vision and Image Understanding, 2016
- Survey on speech emotion recognition: Features, classification schemes, and databasesPattern Recognition, 2011
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008
- Recognizing action units for facial expression analysisIEEE Transactions on Pattern Analysis and Machine Intelligence, 2001
- Measuring emotion: The self-assessment manikin and the semantic differentialJournal of Behavior Therapy and Experimental Psychiatry, 1994
- Constants across cultures in the face and emotion.Journal of Personality and Social Psychology, 1971