Speech Emotion Recognition Based on Self-Attention Weight Correction for Acoustic and Text Features
Open Access
- 3 November 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 10, 115732-115743
Abstract
Speech emotion recognition (SER) is essential for understanding a speaker’s intention. Recently, some groups have attempted to improve SER performance using a bidirectional long short-term memory (BLSTM) to extract features from speech sequences and a self-attention mechanism to focus on the important parts of the speech sequences. SER also benefits from combining the information in speech with text, which can be accomplished automatically using an automatic speech recognizer (ASR), further improving its performance. However, ASR performance deteriorates in the presence of emotion in speech. Although there is a method to improve ASR performance in the presence of emotional speech, it requires the fine-tuning of ASR, which has a high computational cost and leads to the loss of cues important for determining the presence of emotion in speech segments, which can be helpful in SER. To solve these problems, we propose a BLSTM-and-self-attention-based SER method using self-attention weight correction (SAWC) with confidence measures. This method is applied to acoustic and text feature extractors in SER to adjust the importance weights of speech segments and words with a high possibility of ASR error. Our proposed SAWC reduces the importance of words with speech recognition error in the text feature while emphasizing the importance of speech segments containing these words in acoustic features. Our experimental results on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset reveal that our proposed method achieves a weighted average accuracy of 76.6%, outperforming other state-of-the-art methods. Furthermore, we investigated the behavior of our proposed SAWC in each of the feature extractors.Keywords
Funding Information
- JST SPRING (JPMJSP2124)
This publication has 37 references indexed in Scilit:
- Automatic speech emotion recognition using recurrent neural networks with local attention2017
- Speech emotion recognition using convolutional and Recurrent Neural Networks2016
- Towards real-time Speech Emotion Recognition using deep neural networks2015
- Librispeech: An ASR corpus based on public domain audio books2015
- Learning Salient Features for Speech Emotion Recognition Using Convolutional Neural NetworksIEEE Transactions on Multimedia, 2014
- Acoustic emotion recognition: A benchmark comparison of performances2009
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008
- Speech Emotion Recognition in E-learning System Based on Affective Computing2007
- Confidence measures for speech recognition: A surveySpeech Communication, 2005
- Emotion detection in task-oriented spoken dialogues2003