JSQA: Speech Quality Assessment with Perceptually-Inspired Contrastive Pretraining Based on JND Audio Pairs
Preprint
Abstract
Speech quality assessment (SQA) is often used to learn a mapping from a high-dimensional input space to a scalar that represents the mean opinion score (MOS) of the perceptual speech quality. Learning such a mapping is challenging for many reasons, but largely because MOS exhibits high levels of inherent variance due to perceptual and experimental-design differences. Many solutions have been proposed, but many approaches do not properly incorporate perceptual factors into their learning algorithms (beyond the MOS label), which could lead to unsatisfactory results. To this end, we propose JSQA, a two-stage framework that pretrains an audio encoder using perceptually-guided contrastive learning on just noticeable difference (JND) pairs, followed by fine-tuning for MOS prediction. We first generate pairs of audio data within JND levels, which are then used to pretrain an encoder to leverage perceptual quality similarity information and map it into an embedding space. The JND pairs come from clean LibriSpeech utterances that are mixed with background noise from CHiME-3, at different signal-to-noise ratios (SNRs). The encoder is later fine-tuned with audio samples from the NISQA dataset for MOS prediction. Experimental results suggest that perceptually-inspired contrastive pretraining significantly improves the model performance evaluated by various metrics when compared against the same network trained from scratch without pretraining. These findings suggest that incorporating perceptual factors into pretraining greatly contributes to the improvement in performance for SQA.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- From the perspective of perceptual speech quality: The robustness of frequency bands to noiseThe Journal of the Acoustical Society of America, 2024
- Exploring the influence of fine-tuning data on wav2vec 2.0 model for blind speech quality prediction2022
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden UnitsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets2021
- MetricNet: Towards Improved Modeling For Non-Intrusive Speech Quality Assessment2021
- A Differentiable Perceptual Audio Metric Learned from Just Noticeable Differences2020
- Quality-Net: An End-to-End Non-intrusive Speech Quality Assessment Model Based on BLSTM2018
- ViSQOL: an objective speech quality modelEURASIP Journal on Audio, Speech, and Music Processing, 2015
- The Just-Noticeable Difference in Speech-to-Noise RatioTrends in Hearing, 2015
- P.563—The ITU-T Standard for Single-Ended Speech Quality AssessmentIEEE Transactions on Audio, Speech, and Language Processing, 2006