Speech Swin-Transformer: Exploring a Hierarchical Transformer with Shifted Windows for Speech Emotion Recognition
Authors:
- 14 April 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 11646-11650
Abstract
Swin-Transformer has demonstrated remarkable success in computer vision by leveraging its hierarchical feature representation based on Transformer. In speech signals, emotional information is distributed across different scales of speech features, e. g., word, phrase, and utterance. Drawing above inspiration, this paper presents a hierarchical speech Transformer with shifted windows to aggregate multi-scale emotion features for speech emotion recognition (SER), called Speech Swin-Transformer. Specifically, we first divide the speech spectrogram into segment-level patches in the time domain, composed of multiple frame patches. These segment-level patches are then encoded using a stack of Swin blocks, in which a local window Transformer is utilized to explore local inter-frame emotional information across frame patches of each segment patch. After that, we also design a shifted window Transformer to compensate for patch correlations near the boundaries of segment patches. Finally, we employ a patch merging operation to aggregate segment-level emotional features for hierarchical speech representation by expanding the receptive field of Transformer from frame-level to segment-level. Experimental results demonstrate that our proposed Speech Swin-Transformer outperforms the state-of-the-art methods.Keywords
Funding Information
- China Postdoctoral Science Foundation
This publication has 19 references indexed in Scilit:
- Time-Frequency Transformer: A Novel Time Frequency Joint Learning Method for Speech Emotion Recognition2023
- RETRACTED ARTICLE: Speech Emotion Recognition: A Comprehensive SurveyWireless Personal Communications, 2023
- Activation functions in deep learning: A comprehensive survey and benchmarkNeurocomputing, 2022
- A systematic literature review of speech emotion recognition approachesNeurocomputing, 2022
- Neural collapse under cross-entropy lossApplied and Computational Harmonic Analysis, 2022
- A Novel end-to-end Speech Emotion Recognition Network with Stacked Transformer Layers2021
- A Comprehensive Review of Speech Emotion Recognition SystemsIEEE Access, 2021
- Databases, features and classifiers for speech emotion recognition: a reviewInternational Journal of Speech Technology, 2018
- Deep neural networks for acoustic emotion recognition: Raising the benchmarks2011
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008