Mel-MViTv2: Enhanced Speech Emotion Recognition With Mel Spectrogram and Improved Multiscale Vision Transformers
Open Access
- 2 October 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 11, 108571-108579
Abstract
Speech emotion recognition aims to automatically identify and classify emotions from speech signals. It plays a crucial role in various applications such as human-computer interaction, affective computing, and social robotics. Over the years, researchers have proposed different approaches for speech emotion recognition, leveraging various classifiers and features. However, despite the advancements, existing methods in speech emotion recognition still have certain limitations. Some approaches rely on handcrafted features that may not capture the full complexity of emotional information present in speech signals, while others may suffer from a lack of robustness and generalization when applied to different datasets. To address these challenges, this paper proposes a speech emotion recognition method that combines Mel spectrogram with Short-Term Fourier Transform (Mel-STFT) and the Improved Multiscale Vision Transformers (MViTv2). The Mel-STFT spectrograms capture both the frequency and temporal information of speech signals, providing a more comprehensive representation of the emotional content. The MViTv2 classifier introduces multi-scale visual modeling with different stages and pooling attention mechanisms. MViTv2 incorporates relative positional embeddings and a residual pooling connection to effectively model the interactions between tokens in the space-time structure, preserve essential information, and improve the efficiency of the model. Experimental results demonstrate that the proposed method generalizes well on different datasets, achieving an accuracy of 91.51% on the Emo-DB dataset, 81.75% on the RAVDESS dataset, and 64.03% on the IEMOCAP dataset.Keywords
Funding Information
- Telekom Malaysia Berhad’s Research and Development (RDTC/231075, RDTC/221064)
- deanship of scientific research, King Khalid University, Saudi Arabia (RGP2/332/44)
This publication has 15 references indexed in Scilit:
- A Novel Heterogeneous Parallel Convolution Bi-LSTM for Speech Emotion RecognitionApplied Sciences, 2021
- Deep scattering network for speech emotion recognition2021
- Convolutional Gated Recurrent Units (CGRU) for Emotion Recognition in Odia Language2021
- Speech Emotion Recognition with a ResNet-CNN-Transformer Parallel Neural Network2021
- Deep Learning Techniques for Speech Emotion Recognition: A Review2019
- Transfer Learning for Improving Speech Emotion Classification Accuracy2018
- The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American EnglishPLOS ONE, 2018
- Spectrogram based multi-task audio classificationMultimedia Tools and Applications, 2017
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008
- A database of German emotional speech2005