Multimodal Emotion Recognition Calibration in Conversations
Open Access
- 28 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 9621-9630
Abstract
Multimodal Emotion Recognition in Conversations (MERC) aims to identify the emotions conveyed by each utterance in a conversational video. Current efforts focus on modeling speaker-sensitive context dependencies and multimodal fusion. Despite the progress, the reliability of MERC methods remains largely unexplored. Extensive empirical studies reveal that current methods suffer from unreliable predictive confidence. Specifically, in some cases, the confidence estimated by these models increases when a modality or specific contextual cues are corrupted, defining these as uncertain samples. This contradicts the foundational principle in informatics, namely, the elimination of uncertainty. Based on this, we propose a novel calibration framework CMERC to calibrate MERC models without altering the model structure. It integrates curriculum learning to guide the model in progressively learning more uncertain samples; hybrid supervised contrastive learning to refine utterance representations, by pulling uncertain samples and others apart; and confidence constraint to penalize the model on uncertain samples. Experimental results on two datasets demonstrate the effectiveness and generalization capabilities of our CMERC across various MERC models, surpassing state-of-the-art methods.Keywords
Funding Information
- Shenzhen Foundational Research Funding (JCYJ20220818102415032)
- The Major Key Project of PCL (PCL2023A09)
- Natural Science Foundation of Guangdong (2023A1515012922)
- National Natural Science Foundation of China (62176076)
- Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies (2022B1212010005k)
This publication has 31 references indexed in Scilit:
- Multimodal Knowledge-enhanced Interactive Network with Mixed Contrastive Learning for Emotion Recognition in ConversationNeurocomputing, 2024
- Rethinking Confidence Calibration for Failure Prediction2022
- Dynamic interactive multiview memory network for emotion recognition in conversationInformation Fusion, 2022
- The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration2022
- Exploration meets exploitation: Multitask learning for emotion recognition based on discrete and dimensional modelsKnowledge-Based Systems, 2022
- BiERU: Bidirectional emotional recurrent unit for conversational sentiment analysisNeurocomputing, 2022
- Improving Calibration for Long-Tailed Recognition2021
- Improving Confidence Estimates for Unfamiliar Examples2020
- Regularizing Class-Wise Predictions via Self-Knowledge Distillation2020
- Silhouettes: A graphical aid to the interpretation and validation of cluster analysisJournal of Computational and Applied Mathematics, 1987