Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimodal Emotion Recognition
Preprint
Abstract
Multimodal emotion recognition aims to recognize emotions for each utterance of multiple modalities, which has received increasing attention for its application in human-machine interaction. Current graph-based methods fail to simultaneously depict global contextual features and local diverse uni-modal features in a dialogue. Furthermore, with the number of graph layers increasing, they easily fall into over-smoothing. In this paper, we propose a method for joint modality fusion and graph contrastive learning for multimodal emotion recognition (Joyful), where multimodality fusion, contrastive learning, and emotion recognition are jointly optimized. Specifically, we first design a new multimodal fusion mechanism that can provide deep interaction and fusion between the global contextual and uni-modal specific features. Then, we introduce a graph contrastive learning framework with inter-view and intra-view contrastive losses to learn more distinguishable representations for samples with different sentiments. Extensive experiments on three benchmark datasets indicate that Joyful achieved state-of-the-art (SOTA) performance compared to all baselines.Keywords
All Related Versions
This publication has 67 references indexed in Scilit:
- 3MASSIV: Multilingual, Multimodal and Multi-Aspect dataset of Social Media Short Videos2022
- Multimodal Token Fusion for Vision Transformers2022
- Recognizing Emotion Cause in ConversationsCognitive Computation, 2021
- Progressive Modality Reinforcement for Human Multimodal Emotion Recognition from Unaligned Multimodal Sequences2021
- Influence of multiple hypothesis testing on reproducibility in neuroimaging research: A simulation study and Python-based softwareJournal of Neuroscience Methods, 2020
- Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent AdvancesIEEE Access, 2019
- OpenFace 2.0: Facial Behavior Analysis Toolkit2018
- Densely Connected Convolutional Networks2017
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008
- A New Status Index Derived from Sociometric AnalysisPsychometrika, 1953