Investigating Polyglot Speech Foundation Models for Learning Collective Emotion from Crowds
Authors:
- Orchid Chetia PhukanOPOrchid Chetia Phukan
- GirishGIGirish
Preprint
Abstract
This paper investigates the polyglot (multilingual) speech foundation models (SFMs) for Crowd Emotion Recognition (CER). We hypothesize that polyglot SFMs, pre-trained on diverse languages, accents, and speech patterns, are particularly adept at navigating the noisy and complex acoustic environments characteristic of crowd settings, thereby offering a significant advantage for CER. To substantiate this, we perform a comprehensive analysis, comparing polyglot, monolingual, and speaker recognition SFMs through extensive experiments on a benchmark CER dataset across varying audio durations (1 sec, 500 ms, and 250 ms). The results consistently demonstrate the superiority of polyglot SFMs, outperforming their counterparts across all audio lengths and excelling even with extremely short-duration inputs. These findings pave the way for adaptation of SFMs in setting up new benchmarks for CER.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Towards A Robust Group-level Emotion Recognition via Uncertainty-Aware LearningIEEE Transactions on Affective Computing, 2025
- Whisper Features for Dysarthric Severity-Level Classification2023
- Evaluating Self-Supervised Speech Representations for Speech Emotion RecognitionIEEE Access, 2022
- XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale2022
- Investigation of Ensemble features of Self-Supervised Pretrained Models for Automatic Speech Recognition2022
- A Self-Fusion Network Based on Contrastive Learning for Group Emotion RecognitionIEEE Transactions on Computational Social Systems, 2022
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech ProcessingIEEE Journal of Selected Topics in Signal Processing, 2022
- Unispeech-Sat: Universal Speech Representation Learning With Speaker Aware Pre-Training2022
- Non-volume preserving-based fusion to group-level emotion recognition on crowd videosPattern Recognition, 2022
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden UnitsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021