Unimodal-driven Distillation in Multimodal Emotion Recognition with Dynamic Fusion
Authors:
- Jiagen LiJLJiagen Li
- Rui YuRYRui Yu
- Huihao HuangHHHuihao Huang
- Huaicheng YanHuaicheng Yan
Preprint
Abstract
Multimodal Emotion Recognition in Conversations (MERC) identifies emotional states across text, audio and video, which is essential for intelligent dialogue systems and opinion analysis. Existing methods emphasize heterogeneous modal fusion directly for cross-modal integration, but often suffer from disorientation in multimodal learning due to modal heterogeneity and lack of instructive guidance. In this work, we propose SUMMER, a novel heterogeneous multimodal integration framework leveraging Mixture of Experts with Hierarchical Cross-modal Fusion and Interactive Knowledge Distillation. Key components include a Sparse Dynamic Mixture of Experts (SDMoE) for capturing dynamic token-wise interactions, a Hierarchical Cross-Modal Fusion (HCMF) for effective fusion of heterogeneous modalities, and Interactive Knowledge Distillation (IKD), which uses a pre-trained unimodal teacher to guide multimodal fusion in latent and logit spaces. Experiments on IEMOCAP and MELD show SUMMER outperforms state-of-the-art methods, particularly in recognizing minority and semantically similar emotions.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in ConversationsIEEE Transactions on Multimedia, 2023
- EmoCaps: Emotion Capsule based Model for Conversational Emotion Recognition2022
- Leveraging recent advances in deep learning for audio-Visual emotion recognitionPattern Recognition Letters, 2021
- CTNet: Conversational Transformer Network for Emotion RecognitionIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation2021
- DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation2019
- MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations2019
- Multimodal sentiment analysis using hierarchical fusion with context modelingKnowledge-Based Systems, 2018
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008