PAD-Guided Multimodal Hybrid Contrastive Emotion Recognition upon STEM-E2VA Dataset
Open Access
- 2 April 2026
- journal article
- research article
- Published by MDPI AG in Multimodal Technologies and Interaction
- Vol. 10 (4) , 38
Abstract
There are still challenges in speech emotion recognition, as the representation capability of single-modal information is limited, there are difficulties in capturing continuous emotional transitions in discrete emotion annotations, and the issues of modal structural differences and cross-sample alignment in multimodal fusion methods persist. To address these, this study undertakes work from both data and model perspectives. For data, a Chinese multimodal database STEM-E2VA was constructed, synchronously collecting four modalities of data: articulatory kinematics, acoustics, glottal signals, and videos. This covers seven discrete emotion categories and employs PAD continuous annotation. By integrating discrete and continuous dimensional annotations, it better represents the distinction between strong and weak emotions under the same discrete emotion label. Concurrently, to process the biases in PAD annotations, we employed the SCL-90 psychological questionnaire to analyze annotators’ cognitive and emotional perceptions, thereby ensuring data reliability. For model, this paper proposes a multimodal supervised contrastive fusion network incorporating PAD perception. It employs a PAD-enhanced hybrid contrastive loss function to optimize intra-model and inter-modal feature alignment. Utilizing a cross-attention mechanism combined with a GRU–Transformer network for temporal feature extraction, it achieves deep fusion of multimodal information, reducing inter-modal discrepancies and cross-class confusion. Experiments demonstrate that the proposed method achieves 85.47% accuracy in discrete sentiment recognition on STEM-E2VA, with a substantial reduction in RMSE for PAD dimension prediction. It also exhibits excellent generalization capability on IEMOCAP, providing a novel framework for integrating discrete and continuous sentiment representations.Keywords
Funding Information
- Youth Fund of the National Natural Science Foundation of China (12004275)
- Shanxi Scholarship Council of China (2024-060)
- Natural Science Foundation of Shanxi Province, China (202403021211098)
- Startup Fund of Shanxi University of Electronic Science and Technology (2025KJ016)
This publication has 35 references indexed in Scilit:
- Using transformers for multimodal emotion recognition: Taxonomies and state of the art reviewEngineering Applications of Artificial Intelligence, 2024
- Multimodal transformer with adaptive modality weighting for multimodal sentiment analysisNeurocomputing, 2023
- Modeling public mood and emotion: Blog and news sentiment and socio-economic phenomenaFuture Generation Computer Systems, 2019
- A review of affective computing: From unimodal analysis to multimodal fusionInformation Fusion, 2017
- Measures of Vocal Attack Time for Healthy Young AdultsJournal of Voice, 2011
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008
- The role of voice quality in communicating emotion, mood and attitudeSpeech Communication, 2003
- Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in TemperamentCurrent Psychology, 1996
- Induction of mood and mood shiftJournal of Clinical Psychology, 1980
- Evidence for a three-factor theory of emotionsJournal of Research in Personality, 1977