Harnessing Multimodal Unlabeled Data for Enhanced Speech Emotion Recognition
- 24 December 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Affective Computing
- Vol. 17 (1) , 1134-1146
Abstract
Speech emotion recognition (SER) often faces challenges due to the lack of large, annotated datasets. The presence of abundance unlabeled data offers a chance to explore methods that could significantly improve SER systems. This study explores the feasibility of enhancing general speech models by incorporating unimodal and multimodal training objectives derived from unlabeled data, specifically tailored to extract emotional content. These multimodal objectives aim to refine self-supervised learning (SSL)-based representations that, while effective in SER, were not originally created to extract emotional cues from speech. Our methodology introduces a set of multimodal objectives focused on capturing information from three primary sources: acoustic signals, through a representation objective based on the extended Geneva Minimalistic Acoustic Parameter Set (eGEMAPS); facial expressions, via visual representations obtained from a pre-trained facial expression recognition system; and textual content, through pseudo-labels generated by a pre-trained emotion sentiment model. These objectives are automatically generated from 70.7 hours of unlabeled emotional content captured in naturalistic settings. We apply our strategy to four state-of-the-art SSL-based speech models, aiming to enhance their capabilities in SER tasks with multimodal signals while still keeping inference strictly audio-only. Our experimental evaluations across the CREMA-D, MSP-IMPROV, and MSP-Podcast datasets demonstrate that our approach significantly improves SER performance, especially in settings with limited labeled data.Keywords
This publication has 67 references indexed in Scilit:
- Improving speech depression detection using transfer learning with wav2vec 2.0 in low-resource environmentsScientific Reports, 2024
- Speech emotion recognition using machine learning — A systematic reviewIntelligent Systems with Applications, 2023
- Leveraging recent advances in deep learning for audio-Visual emotion recognitionPattern Recognition Letters, 2021
- Active Learning for Speech Emotion Recognition Using Deep Neural Network2019
- Multimodal sentiment analysis using hierarchical fusion with context modelingKnowledge-Based Systems, 2018
- Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network2016
- Combining feature-level and decision-level fusion in a hierarchical classifier for emotion recognition in the wildJournal on Multimodal User Interfaces, 2015
- A personalized emotion recognition system using an unsupervised feature adaptation scheme2012
- Unsupervised learning in cross-corpus acoustic emotion recognition2011
- Vocal communication of emotion: A review of research paradigmsSpeech Communication, 2003