Cross-Modal Fusion Techniques for Utterance-Level Emotion Recognition from Text and Speech
Authors:
- 4 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-5
Abstract
Multimodal emotion recognition (MER) is a fundamental complex research problem due to the uncertainty of human emotional expression and the heterogeneity gap between different modalities. Audio and text modalities are particularly important for a human participant in understanding emotions. Although many successful attempts have been designed multimodal representations for MER, there still exist multiple challenges to be addressed: 1) bridging the heterogeneity gap between multimodal features and model inter- and intramodal interactions of multiple modalities; 2) effectively and efficiently modeling the contextual dynamics in the conversation sequence. In this paper, we propose Cross-Modal RoBERTa (CM-RoBERTa) model for emotion detection from spoken audio and corresponding transcripts. As the core unit of the CM-RoBERTa, parallel self- and cross- attention is designed to dynamically capture inter- and intra-modal interactions of audio and text. Specially, the mid-level fusion and residual module are employed to model longterm contextual dependencies and learn modality-specific patterns. We evaluate the approach on the MELD dataset and the experimental results show the proposed approach achieves the state-of-art performance on the dataset.Keywords
All Related Versions
Funding Information
- Queen Mary University of London
This publication has 21 references indexed in Scilit:
- Fusion of deep learning features with mixture of brain emotional learning for audio-visual emotion recognitionSpeech Communication, 2021
- Cross-Subject Multimodal Emotion Recognition Based on Hybrid FusionIEEE Access, 2020
- A Review of Generalizable Transfer Learning in Automatic Emotion RecognitionFrontiers in Computer Science, 2020
- Speech Emotion Recognition Using Deep Learning Techniques: A ReviewIEEE Access, 2019
- Look, Listen, and Learn More: Design Choices for Deep Audio Embeddings2019
- Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis2019
- Emotion detection from text and speech: a surveySocial Network Analysis and Mining, 2018
- ICON: Interactive Conversational Memory Network for Multimodal Emotion Detection2018
- Opensmile2010
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008