Improvement of Multimodal Emotion Recognition Based on Temporal-Aware Bi-Direction Multi-Scale Network and Multi-Head Attention Mechanisms
Open Access
- 13 April 2024
- journal article
- research article
- Published by MDPI AG in Applied Sciences
- Vol. 14 (8) , 3276
Abstract
Emotion recognition is a crucial research area in natural language processing (NLP), aiming to identify emotional states such as happiness, anger, and sadness from various sources like speech, text, and facial expressions. In this paper, we propose an improved MMER (multimodal emotion recognition) method using TIM-Net (Temporal-Aware Bi-Direction Multi-Scale Network) and attention mechanisms. Firstly, we introduce the methods for extracting and fusing the multimodal features. Then, we present the TIM-Net and attention mechanisms, which are utilized to enhance the MMER algorithm. We evaluate our approach on the IEMOCAP and MELD datasets, and compared to existing methods, our approach demonstrates superior performance. The weighted accuracy recall (WAR) on the IEMOCAP dataset is 83.9%, and the weighted accuracy recall rate on the MELD dataset is 62.7%. Finally, the impact of the TIM-Net model and the attention mechanism on the emotion recognition performance is further investigated through ablation experiments.Keywords
Funding Information
- National Key Research and Development Program of China (2021YFF0603904, ZJ2022-004)
- Fundamental Research Funds for the Central Universities (2021YFF0603904, ZJ2022-004)
This publication has 58 references indexed in Scilit:
- A CNN-LSTM-Based Hybrid Deep Learning Approach for Sentiment Analysis on Monkeypox TweetsNew Generation Computing, 2023
- Identifying emotions from facial expressions using a deep convolutional neural network-based approachMultimedia Tools and Applications, 2023
- Facial emotion recognition and music recommendation system using CNN-based deep learning techniquesEvolving Systems, 2023
- Multimodal sentiment analysis based on fusion methods: A surveyInformation Fusion, 2023
- Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directionsInformation Fusion, 2022
- Spatiotemporal causal convolutional network for forecasting hourly PM2.5 concentrations in Beijing, ChinaComputers & Geosciences, 2021
- Multistage classification scheme to enhance speech emotion recognitionInternational Journal of Speech Technology, 2019
- On-line emotion recognition in a 3-D activation-valence-time continuum using acoustic and linguistic cuesJournal on Multimodal User Interfaces, 2009
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008
- Term-weighting approaches in automatic text retrievalInformation Processing & Management, 1988