Speech Emotion Recognition Based on Parallel CNN-Attention Networks with Multi-Fold Data Augmentation
Open Access
- 28 November 2022
- journal article
- research article
- Published by MDPI AG in Electronics
- Vol. 11 (23) , 3935
Abstract
In this paper, an automatic speech emotion recognition (SER) task of classifying eight different emotions was experimented using parallel based networks trained using the Ryeson Audio-Visual Dataset of Speech and Song (RAVDESS) dataset. A combination of a CNN-based network and attention-based networks, running in parallel, was used to model both spatial features and temporal feature representations. Multiple Augmentation techniques using Additive White Gaussian Noise (AWGN), SpecAugment, Room Impulse Response (RIR), and Tanh Distortion techniques were used to augment the training data to further generalize the model representation. Raw audio data were transformed into Mel-Spectrograms as the model’s input. Using CNN’s proven capability in image classification and spatial feature representations, the spectrograms were treated as an image with the height and width represented by the spectrogram’s time and frequency scales. Temporal feature representations were represented by attention-based models Transformer, and BLSTM-Attention modules. Proposed architectures of the parallel CNN-based networks running along with Transformer and BLSTM-Attention modules were compared with standalone CNN architectures and attention-based networks, as well as with hybrid architectures with CNN layers wrapped in time-distributed wrappers stacked on attention-based networks. In these experiments, the highest accuracy of 89.33% for a Parallel CNN-Transformer network and 85.67% for a Parallel CNN-BLSTM-Attention Network were achieved on a 10% hold-out test set from the dataset. These networks showed promising results based on their accuracies, while keeping significantly less training parameters compared with non-parallel hybrid models.Keywords
Funding Information
- Ministry of Trade, Industry and Energy (20012603)
This publication has 42 references indexed in Scilit:
- Spectrogram based multi-task audio classificationMultimedia Tools and Applications, 2017
- Emotion Recognition in Sound2017
- Automatic speech emotion recognition using recurrent neural networks with local attention2017
- Speech emotion recognition with deep learning2017
- Emotion Communication SystemIEEE Access, 2016
- Data Augmentation for Deep Neural Network Acoustic ModelingIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2015
- Implementation and Comparison of Speech Emotion Recognition System Using Gaussian Mixture Model (GMM) and K- Nearest Neighbor (K-NN) TechniquesProcedia Computer Science, 2015
- Speech Emotion Recognition under White NoiseArchives of Acoustics, 2013
- Speech Emotion Recognition Using Gaussian Mixture Model2012
- Emotion recognition in human-computer interactionIEEE Signal Processing Magazine, 2001