CRNet: Context-Guided Recovering Network for Incomplete Multimodal Emotion Recognition in Conversation
- 1 June 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. PP (15209210) , 1-15
Abstract
Incomplete multimodal learning has become a critical challenge in Emotion Recognition in Conversation (ERC). However, most methods focus on individual utterances instead of conversational data, thus often overlooking the rich contextual information embedded within conversations. Moreover, existing incomplete ERC approaches are typically limited to joint multimodal representation learning for missing modality recovery, failing to account for the complementary information inherent in modality-specific characteristics. In this paper, we propose a Context-guided Recovering Network (CRNet) to comprehensively explore both modality characteristics and contextual information in conversational data for incomplete multimodal ERC. Specifically, to effectively leverage cross-modal semantic consistency while preserving the modality-specific information, we design a context consistent recovery module. This module uses the semantic relations between utterances to guide the recovery of missing components. Furthermore, to maintain the balance between preserving emotion consistency and uncovering emotion shifts in contextual modeling, we introduce a dual-level semantic learning module, which is composed of the instance-level contrastive unit and label-based emotion shift unit. The instance-level contrastive unit enforces similarity constraints across recovered instances within each modality, while the label-based emotion shift unit captures emotional variations in multimodal fusion representations. Experimental results on four benchmark datasets demonstrate that the proposed CRNet achieves remarkable performance compared with state-of-the-art baselines.Keywords
Funding Information
- NSFC Fund (62572145)
- Shenzhen Key Technical Project (JSGG20220831092805009)
- Shenzhen Science and Technology Program (KJZD20230923114600002)
- Guangdong Natural Science Fund Project (2026A1515011491)
This publication has 58 references indexed in Scilit:
- SDR-GNN: Spectral Domain Reconstruction Graph Neural Network for incomplete multimodal learning in conversational emotion recognitionKnowledge-Based Systems, 2024
- Multi-modal graph context extraction and consensus-aware learning for emotion recognition in conversationKnowledge-Based Systems, 2024
- Decoupled Multimodal Distilling for Emotion Recognition2023
- Multivariate, Multi-Frequency and Multimodal: Rethinking Graph Neural Networks for Emotion Recognition in Conversation2023
- Emotion recognition in conversations with emotion shift detection based on multi-task learningKnowledge-Based Systems, 2022
- Modeling Relational Data with Graph Convolutional Networks2018
- Missing Modalities Imputation via Cascaded Residual Autoencoder2017
- Emotion Recognition in Affective Tutoring Systems: Collection of Ground-truth DataProcedia Computer Science, 2017
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008
- Relations Between Two Sets of Variates1992