GEmo-CLAP: Gender-Attribute-Enhanced Contrastive Language-Audio Pretraining for Speech Emotion Recognition
Authors:
Preprint
Abstract
Contrastive learning based cross-modality pretraining methods have recently exhibited impressive success in diverse fields. In this paper, we propose GEmo-CLAP, a kind of gender-attribute-enhanced contrastive language-audio pretraining (CLAP) method for speech emotion recognition. Specifically, a novel emotion CLAP model (Emo-CLAP) is first built, utilizing various self-supervised pre-trained models. Second, considering the importance of gender attribute in speech emotion modeling, the soft label based GEmo-CLAP (SL-GEmo-CLAP) and multi-task learning based GEmo-CLAP (ML-GEmo-CLAP) are further proposed to integrate the emotion and gender information of speech signals, forming more reasonable objectives. Extensive experiments on IEMOCAP show that our proposed two GEmo-CLAP models consistently outperform the baseline Emo-CLAP with different pre-trained models, while also achieving the best recognition performance compared with recent state-of-the-art methods. Noticeably, the proposed WavLM-based ML-GEmo-CLAP obtains the best UAR of 80.16\% and WAR of 82.06\%.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Temporal Modeling Matters: A Novel Temporal Emotional Modeling Approach for Speech Emotion Recognition2023
- Dawn of the Transformer Era in Speech Emotion Recognition: Closing the Valence GapIEEE Transactions on Pattern Analysis and Machine Intelligence, 2023
- Multi-level Fusion of Wav2vec 2.0 and BERT for Multimodal Emotion Recognition2022
- Multi-Corpus Speech Emotion Recognition for Unseen Corpus Using Corpus-Wise Weights in Classification Loss2022
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech ProcessingIEEE Journal of Selected Topics in Signal Processing, 2022
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden UnitsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- Emotion Recognition from Speech Using wav2vec 2.0 Embeddings2021
- Speech Technology for Healthcare: Opportunities, Challenges, and State of the ArtIEEE Reviews in Biomedical Engineering, 2020
- Customer Satisfaction Estimation in Contact Center Calls Based on a Hierarchical Multi-Task ModelIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020