A Study On Incorporating Whisper For Robust Speech Assessment
Authors:
- 15 July 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-6
Abstract
This research introduces an enhanced version of the multi-objective speech assessment model–MOSA-Net+, by leveraging the acoustic features from Whisper, a large-scaled weakly supervised model. We first investigate the effectiveness of Whisper in deploying a more robust speech assessment model. After that, we explore combining representations from Whisper and SSL models. The experimental results reveal that Whisper’s embedding features can contribute to more accurate prediction performance. Moreover, combining the embedding features from Whisper and SSL models only leads to marginal improvement. As compared to intrusive methods, MOSA-Net, and other SSL-based speech assessment models, MOSA-Net+ yields notable improvements in estimating subjective quality and intelligibility scores across all evaluation metrics in Taiwan Mandarin Hearing In Noise test - Quality & Intelligibility (TMHINT-QI) dataset. To further validate its robustness, MOSA-Net+ was tested in the noisy-and-enhanced track of the VoiceMOS Challenge 2023, where it obtained the top-ranked performance among nine systems.Keywords
All Related Versions
This publication has 31 references indexed in Scilit:
- KAQ: A Non-Intrusive Stacking Framework for Mean Opinion Score Prediction with Multi-Task Learning2023
- The Voicemos Challenge 2023: Zero-Shot Subjective Speech Quality Prediction for Multiple Domains2023
- Generalization Ability of MOS Prediction Networks2022
- LDNet: Unified Listener Dependent Modeling in MOS Prediction for Synthetic Speech2022
- An Attention Enhanced Multi-Task Model for Objective Speech Assessment in Real-World Environments2020
- T-GSA: Transformer with Gaussian-Weighted Self-Attention for Speech Enhancement2020
- Speaker Recognition from Raw Waveform with SincNet2018
- Raw waveform-based speech enhancement by fully convolutional networks2017
- Speech intelligibility prediction using a Neurogram Similarity Index MeasureSpeech Communication, 2011
- An adaptive KLT approach for speech enhancementIEEE Transactions on Speech and Audio Processing, 2001