Semi-Supervised Cognitive State Classification from Speech with Multi-View Pseudo-Labeling
Authors:
- 6 April 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-5
Abstract
The lack of labeled data is a common challenge in speech classification tasks, particularly those requiring extensive subjective assessment, such as cognitive state classification. In this work, we propose a Semi-Supervised Learning (SSL) framework, introducing a novel multi-view pseudo-labeling method that leverages both acoustic and linguistic characteristics to select the most confident data for training the classification model. Acoustically, unlabeled data are compared to labeled data using the Fréchet audio distance, calculated from embeddings generated by multiple audio encoders. Linguistically, large language models are prompted to revise automatic speech recognition transcriptions and predict labels based on our proposed task-specific knowledge. High-confidence data are identified when pseudo-labels from both sources align, while mismatches are treated as low-confidence data. A bimodal classifier is then trained to iteratively label the low-confidence data until a predefined criterion is met. We evaluate our SSL framework on emotion recognition and dementia detection tasks. Experimental results demonstrate that our method achieves competitive performance compared to fully supervised learning using only 30% of the labeled data and significantly outperforms two selected baselines.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- Generative Speech Recognition Error Correction With Large Language Models and Task-Activating Prompting2023
- Fusing ASR Outputs in Joint Training for Speech Emotion Recognition2022
- Speech Emotion Recognition Using Semi-Supervised Learning with Efficient Labeling Strategies2021
- Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining2021
- Self-Training With Noisy Student Improves ImageNet Classification2020
- Leveraging Unlabeled Data for Emotion Recognition With Enhanced Collaborative Semi-Supervised LearningIEEE Access, 2018
- CNN architectures for large-scale audio classification2017
- Co-training succeeds in Computational Paralinguistics2013
- Semi-supervised learning helps in sound event classification2012
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008