Learning Noise-Robust Joint Representation for Multimodal Emotion Recognition under Incomplete Data Scenarios
Open Access
- 28 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 116-124
Abstract
Multimodal emotion recognition (MER) in practical scenarios is significantly challenged by the presence of missing or incomplete data across different modalities. To overcome these challenges, researchers have aimed to simulate incomplete conditions during the training phase to enhance the system's overall robustness. Traditional methods have often involved discarding data or substituting data segments with zero vectors to approximate these incompletenesses. However, such approaches neither accurately represent real-world conditions nor adequately address the issue of noisy data availability. For instance, a blurry image cannot be simply replaced with zero vectors, while still retaining information. To tackle this issue and develop a more precise MER system, we introduce a novel noise-robust MER model that effectively learns robust multimodal joint representations from noisy data. This approach includes two pivotal components: firstly, a noise scheduler that adjusts the type and level of noise in the data to emulate various realistic incomplete situations. Secondly, a Variational AutoEncoder (VAE)-based module is employed to reconstruct these robust multimodal joint representations from the noisy inputs. Notably, the introduction of the noise scheduler enables the exploration of an entirely new type of incomplete data condition, which is impossible with existing methods. Extensive experimental evaluations on the benchmark datasets IEMOCAP and CMU-MOSEI demonstrate the effectiveness of the noise scheduler and the excellent performance of our proposed model. Our project is publicly available on https://github.com/WooyoohL/Noise-robust_MER.Keywords
All Related Versions
Funding Information
- Young Scientists Fund of the National Natural Science Foundation of China (62206136)
- the Outstanding Youth Foundation of the Natural Science Foundation of Inner Mongolia (2022JQ05)
- the Hohhot Collaborative Innovation Project for Universities and Institutes, the Youth Science and Technology Talent Cultivation Project of Inner Mongolia University (21221505)
- the Science and Technology Program of Inner Mongolia Autonomous Region (2021GG0158)
- Inner Mongolia Science and Technology Achievement Transfer and Transformation Demonstration Zone, University Collaborative Innovation Base, and University Entrepreneurship Training Base (21300-231510)
- the fund of Supporting the Reform and Development of Local Universities (Disciplinary Construction) and the special research project of First-class Discipline of Inner Mongolia A. R. of China under Grant (YLXKZX-ND-036)
- The Guangdong Provincial Key Laboratory of Human Digital Twin (2022B1212010004)
- National Natural Science Foundation of China (62066033)
This publication has 24 references indexed in Scilit:
- Accuracy weighted diversity-based online boostingExpert Systems with Applications, 2020
- A comprehensive survey on impulse and Gaussian denoising filters for digital imagesSignal Processing, 2019
- Impulsive Noise Recovery and Elimination: A Sparse Machine Learning Based ApproachIEEE Transactions on Vehicular Technology, 2019
- Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data2018
- Deep Adversarial Learning for Multi-Modality Missing Data Completion2018
- Densely Connected Convolutional Networks2017
- Deep Residual Learning for Image Recognition2016
- Opensmile2010
- ImageNet: A large-scale hierarchical image database2009
- Adaptive robust impulse noise filteringIEEE Transactions on Signal Processing, 1995