Robust Representation Learning for Multimodal Emotion Recognition with Contrastive Learning and Mixup
Open Access
- 28 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 93-97
Abstract
Multimodal emotion recognition (MER) plays a crucial role in user sentiment analysis and enhancing human-computer interaction experiences. However, in real-world scenarios, noise interference such as environmental noise and image blurring is widespread, impacting the model's ability to recognize emotions. Therefore, improving the robustness of MER models to handle complex and variable noise interference is a significant challenge. To address this challenge, we propose a robust representation learning method based on contrastive learning and Mixup data augmentation strategies to stabilize model performance on noisy data. Specifically, we first perform Mixup data augmentation in the representation space of each modality to broaden class decision boundaries and enhance model generalization. Meanwhile, through a contrastive learning strategy, we align the representations of clean and noisy data to improve model robustness. We evaluate our model's performance on the test set of the MER2024 MER-NOISE track. Our model achieves a weighted average F-score score of 82.71%, which is 3.09% higher than the MER2024 Baseline model. This result demonstrates that our method effectively enhances model robustness and generalization.Keywords
This publication has 19 references indexed in Scilit:
- MusicLDM: Enhancing Novelty in text-to-music Generation Using Beat-Synchronous mixup Strategies2024
- A novel transformer autoencoder for multi-modal emotion recognition with incomplete dataNeural Networks, 2024
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech ProcessingIEEE Journal of Selected Topics in Signal Processing, 2022
- MixAugment & Mixup: Augmentation Methods for Facial Expression Recognition2022
- Emotion recognition using multi-modal data and machine learning techniques: A tutorial and reviewInformation Fusion, 2020
- Unsupervised Pretraining Transfers Well Across Languages2020
- Ensemble Learning for AI Developers2020
- Ensemble learning: A surveyWIREs Data Mining and Knowledge Discovery, 2018
- End-to-End Multimodal Emotion Recognition Using Deep Neural NetworksIEEE Journal of Selected Topics in Signal Processing, 2017
- Robust emotion recognition from low quality and low bit rate video: A deep learning approach2017