HYFuse: Aligning Heterogeneous Speech Pre-Trained Representations in Hyperbolic Space for Speech Emotion Recognition
Authors:
- Orchid Chetia PhukanOPOrchid Chetia Phukan
- GirishGIGirish
Preprint
Abstract
Compression-based representations (CBRs) from neural audio codecs such as EnCodec capture intricate acoustic features like pitch and timbre, while representation-learning-based representations (RLRs) from pre-trained models trained for speech representation learning such as WavLM encode high-level semantic and prosodic information. Previous research on Speech Emotion Recognition (SER) has explored both, however, fusion of CBRs and RLRs haven't been explored yet. In this study, we solve this gap and investigate the fusion of RLRs and CBRs and hypothesize they will be more effective by providing complementary information. To this end, we propose, HYFuse, a novel framework that fuses the representations by transforming them to hyperbolic space. With HYFuse, through fusion of x-vector (RLR) and Soundstream (CBR), we achieve the top performance in comparison to individual representations as well as the homogeneous fusion of RLRs and CBRs and report SOTA.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- EMO-Codec: An In-Depth Look at Emotion Preservation Capacity of Legacy and Neural Codec Models with Subjective and Objective Evaluations2024
- WavLM model ensemble for audio deepfake detection2024
- Investigation of Ensemble of Self-Supervised Models for Speech Emotion Recognition2023
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech ProcessingIEEE Journal of Selected Topics in Signal Processing, 2022
- Shoelace pattern-based speech emotion recognition of the lecturers in distance education: ShoePat23Applied Acoustics, 2022
- SoundStream: An End-to-End Neural Audio CodecIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden UnitsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- Towards a Small Set of Robust Acoustic Features for Emotion Recognition: ChallengesIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2015
- CREMA-D: Crowd-Sourced Emotional Multimodal Actors DatasetIEEE Transactions on Affective Computing, 2014
- A database of German emotional speech2005