Masked Graph Learning With Recurrent Alignment for Multimodal Emotion Recognition in Conversation
- 26 July 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 32 (23299290) , 4298-4312
Abstract
Since Multimodal Emotion Recognition in Conversation (MERC) can be applied to public opinion monitoring, intelligent dialogue robots, and other fields, it has received extensive research attention in recent years. Unlike traditional unimodal emotion recognition, MERC can fuse complementary semantic information between multiple modalities (e.g., text, audio, and vision) to improve emotion recognition. However, previous work ignored the inter-modal alignment process and the intra-modal noise information before multimodal fusion but directly fuses multimodal features, which will hinder the model for representation learning. In this study, we have developed a novel approach called Masked Graph Learning with Recursive Alignment (MGLRA) to tackle this problem, which uses a recurrent iterative module with memory to align multimodal features, and then uses the masked GCN for multimodal feature fusion. First, we employ LSTM to capture contextual information and use a graph attention-filtering mechanism to eliminate noise effectively within the modality. Second, we build a recurrent iteration module with a memory function, which can use communication between different modalities to eliminate the gap between modalities and achieve the preliminary alignment of features between modalities. Then, a cross-modal multi-head attention mechanism is introduced to achieve feature alignment between modalities and construct a masked GCN for multimodal feature fusion, which can perform random mask reconstruction on the nodes in the graph to obtain better node feature representation. Finally, we utilize a multilayer perceptron (MLP) for emotion recognition. Extensive experiments on two benchmark datasets (i.e., IEMOCAP and MELD) demonstrate that MGLRA outperforms state-of-the-art methods. Our code is publicly available at https://github.com/FuchenZhang/MGLRA.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (69189338, 62372478)
- Research Foundation of Education Bureau of Hunan Province of China (22B0275)
- Changsha Natural Science Foundation (kq2202294)
This publication has 37 references indexed in Scilit:
- RBA-GCN: Relational Bilevel Aggregation Graph Convolutional Network for Emotion RecognitionIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023
- Multi-level Attention Map Network for Multimodal Sentiment AnalysisIEEE Transactions on Knowledge and Data Engineering, 2022
- Multimodal Emotion Recognition With Temporal and Semantic ConsistencyIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval2021
- Bridging Text and Video: A Universal Multimodal Transformer for Audio-Visual Scene-Aware DialogIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- CTNet: Conversational Transformer Network for Emotion RecognitionIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- Adapted Dynamic Memory Network for Emotion Recognition in ConversationIEEE Transactions on Affective Computing, 2020
- Speech Emotion Recognition with Local-Global Aware Deep Representation Learning2020
- The Handbook of Conversation Analysis2012
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008