MM-EMOR: Multi-Modal Emotion Recognition of Social Media Using Concatenated Deep Learning Networks
Open Access
- 13 October 2023
- journal article
- research article
- Published by MDPI AG in Big Data and Cognitive Computing
- Vol. 7 (4) , 164
Abstract
Emotion recognition is crucial in artificial intelligence, particularly in the domain of human–computer interaction. The ability to accurately discern and interpret emotions plays a critical role in helping machines to effectively decipher users’ underlying intentions, allowing for a more streamlined interaction process that invariably translates into an elevated user experience. The recent increase in social media usage, as well as the availability of an immense amount of unstructured data, has resulted in a significant demand for the deployment of automated emotion recognition systems. Artificial intelligence (AI) techniques have emerged as a powerful solution to this pressing concern in this context. In particular, the incorporation of multimodal AI-driven approaches for emotion recognition has proven beneficial in capturing the intricate interplay of diverse human expression cues that manifest across multiple modalities. The current study aims to develop an effective multimodal emotion recognition system known as MM-EMOR in order to improve the efficacy of emotion recognition efforts focused on audio and text modalities. The use of Mel spectrogram features, Chromagram features, and the Mobilenet Convolutional Neural Network (CNN) for processing audio data are central to the operation of this system, while an attention-based Roberta model caters to the text modality. The methodology of this study is based on an exhaustive evaluation of this approach across three different datasets. Notably, the empirical findings show that MM-EMOR outperforms competing models across the same datasets. This performance boost is noticeable, with accuracy gains of an impressive 7% on one dataset and a substantial 8% on another. Most significantly, the observed increase in accuracy for the final dataset was an astounding 18%.Keywords
This publication has 33 references indexed in Scilit:
- Visual-audio emotion recognition based on multi-task and ensemble learning with multiple featuresNeurocomputing, 2020
- Multimodal Approach of Speech Emotion Recognition Using Multi-Level Multi-Head Fusion Attention-Based Recurrent Neural NetworkIEEE Access, 2020
- Learning Alignment for Multimodal Emotion Recognition from Speech2019
- Deep Multimodal Representation Learning: A SurveyIEEE Access, 2019
- High Performance Text Recognition Using a Hybrid Convolutional-LSTM Implementation2017
- Learning Salient Features for Speech Emotion Recognition Using Convolutional Neural NetworksIEEE Transactions on Multimedia, 2014
- Dynamic Facial Emotion Recognition Oriented to HCI ApplicationsInteracting with Computers, 2013
- Multimodal fusion for multimedia analysis: a surveyMultimedia Systems, 2010
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008
- Speech emotion recognition using hidden Markov modelsSpeech Communication, 2003