Advancing Spanish Speech Emotion Recognition: A Comprehensive Benchmark of Pre-Trained Models
Authors:
- Alex MaresAMAlex Mares
- Gerardo Diaz-ArangoGDGerardo Diaz-Arango
- Jorge Perez-Jacome-FriscioneJPJorge Perez-Jacome-Friscione
- Hector Vazquez-LealHector Vazquez-Leal
- Luis Hernandez-MartinezLHLuis Hernandez-Martinez
- Jesus Huerta-ChuaJHJesus Huerta-Chua
- Andres Felipe Jaramillo-AlvaradoAJAndres Felipe Jaramillo-Alvarado
- Alfonso Dominguez-ChavezADAlfonso Dominguez-Chavez
Open Access
- 14 April 2025
- journal article
- research article
- Published by MDPI AG in Applied Sciences
- Vol. 15 (8) , 4340
Abstract
Feature extraction for speech emotion recognition (SER) has evolved from handcrafted techniques through deep learning methods to embeddings derived from pre-trained models (PTMs). This study presents the first comparative analysis focused on using PTMs for Spanish SER, evaluating six models—Whisper, Wav2Vec 2.0, WavLM, HuBERT, TRILLsson, and CLAP—across six emotional speech databases: EmoMatchSpanishDB, MESD, MEACorpus, EmoWisconsin, INTER1SP, and EmoFilm. We propose a robust framework combining layer-wise feature extraction with Leave-One-Speaker-Out validation to ensure interpretable model comparisons. Our method significantly outperforms existing state-of-the-art benchmarks, notably achieving scores on metrics such as F1 on EmoMatchSpanishDB (88.32%), INTER1SP (99.83%), and MEACorpus (92.53%). Layer-wise analyses reveal optimal emotional representation extraction at early layers in 24-layer models and middle layers in larger architectures. Additionally, TRILLsson exhibits remarkable generalization in speaker-independent evaluations, highlighting the necessity of strategic model selection, fine-tuning, and language-specific adaptations to maximize SER performance for Spanish.Keywords
Funding Information
- The Secretariat of Science, Humanities, Technology, and Innovation (SECIHTI) (1282871, 480421)
This publication has 76 references indexed in Scilit:
- Rethinking of BERT sentence embedding for text classificationNeural Computing and Applications, 2024
- Spanish MEACorpus 2023: A multimodal speech–text corpus for emotion analysis in Spanish from natural environmentsComputer Standards & Interfaces, 2024
- Machine learning for human emotion recognition: a comprehensive reviewNeural Computing and Applications, 2024
- EmoMatchSpanishDB: study of speech emotion recognition machine learning models in a new Spanish elicited databaseMultimedia Tools and Applications, 2023
- A comprehensive survey on emotion recognition based on electroencephalograph (EEG) signalsMultimedia Tools and Applications, 2023
- A critical review of state‐of‐the‐art chatbot designs and applicationsWIREs Data Mining and Knowledge Discovery, 2021
- A survey of speech emotion recognition in natural environmentDigital Signal Processing, 2021
- Deep learning approaches for speech emotion recognition: state of the art and research challengesMultimedia Tools and Applications, 2021
- Support Vector Machine: Principles, Parameters, and Applications2017
- ICARUS: Source generator based real-time recognition of speech in noisy stressful and Lombard effect environments ☆Speech Communication, 1995