Attention-Based Multi-Learning Approach for Speech Emotion Recognition With Dilated Convolution
Open Access
- 21 November 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 10, 122302-122313
Abstract
The success of deep learning in speech emotion recognition has led to its application in resource-constrained devices. It has been applied in human-to-machine interaction applications like social living assistance, authentication, health monitoring and alertness systems. In order to ensure a good user experience, robust, accurate and computationally efficient deep learning models are necessary. Recurrent neural networks (RNN) like long short-term memory (LSTM), gated recurrent units (GRU) and their variants that operate sequentially are often used to learn time series sequences of the signal, analyze long-term dependencies and the contexts of the utterances in the speech signal. However, due to their sequential operation, they encounter problems in convergence and sluggish training that uses a lot of memory resources and encounters the vanishing gradient problem. In addition, they do not consider spatial cues that may exist in the speech signal. Therefore, we propose an attention-based multi-learning model (ABMD) that uses residual dilated causal convolution (RDCC) blocks and dilated convolution (DC) layers with multi-head attention. The proposed ABMD model achieves comparable performance while taking global contextualized long-term dependencies between features in a parallel manner using a large receptive field with less increase in the number of parameters compared to the number of layers and considers spatial cues among the speech features. Spectral and voice quality features extracted from the raw speech signals are used as inputs. The proposed ABMD model obtained a recognition accuracy and F1 score of 93.75% and 92.50% on the SAVEE datasets, 85.89% and 85.34% on the RAVDESS datasets and 95.93% and 95.83% on the EMODB datasets. The model’s robustness in terms of the confusion ratio of the individual discrete emotions especially happiness which is often confused with emotions that belong to the same dimensional plane with it also improved when validated on the same datasets.Keywords
Funding Information
- Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2021R1A6A1A03043144)
This publication has 36 references indexed in Scilit:
- Emotion Recognition from Raw Speech using Wavenet2019
- Attention-Based Dense LSTM for Speech Emotion RecognitionIEICE Transactions on Information and Systems, 2019
- Dilated Residual Network with Multi-head Self-attention for Speech Emotion Recognition2019
- Multimodal Integration of Emotional Signals from Voice, Body, and Context: Effects of (In)Congruence on Emotion Recognition and Attitudes Towards RobotsInternational Journal of Social Robotics, 2019
- Speech emotion recognition using deep 1D & 2D CNN LSTM networksBiomedical Signal Processing and Control, 2019
- The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American EnglishPLOS ONE, 2018
- Affect representation and recognition in 3D continuous valence–arousal–dominance spaceMultimedia Tools and Applications, 2016
- Effective Approaches to Attention-based Neural Machine Translation2015
- Bidirectional recurrent neural networksIEEE Transactions on Signal Processing, 1997
- Long Short-Term MemoryNeural Computation, 1997