Using RLHF to align speech enhancement approaches to mean-opinion quality scores
Preprint
Abstract
Objective speech quality measures are typically used to assess speech enhancement algorithms, but it has been shown that they are sub-optimal as learning objectives because they do not always align well with human subjective ratings. This misalignment often results in noticeable distortions and artifacts that cause speech enhancement to be ineffective. To address these issues, we propose a reinforcement learning from human feedback (RLHF) framework to fine-tune an existing speech enhancement approach by optimizing performance using a mean-opinion score (MOS)-based reward model. Our results show that the RLHF-finetuned model has the best performance across different benchmarks for both objective and MOS-based speech quality assessment metrics on the Voicebank+DEMAND dataset. Through ablation studies, we show that both policy gradient loss and supervised MSE loss are important for balanced optimization across the different metrics.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- CMGAN: Conformer-Based Metric-GAN for Monaural Speech EnhancementIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024
- Attention-Based Speech Enhancement Using Human Quality Perception ModelingIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023
- DPT-FSNet: Dual-Path Transformer Based Full-Band and Sub-Band Fusion Network for Speech Enhancement2022
- Objective Measures of Perceptual Audio Quality Reviewed: An Evaluation of Their Application Domain DependenceIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- A Pyramid Recurrent Network for Predicting Crowdsourced Speech-Quality Ratings of Real-World Signals2020
- Intrusive and Non-Intrusive Perceptual Speech Quality Assessment Using a Convolutional Neural Network2019
- End-to-End Waveform Utterance Enhancement for Direct Evaluation Metrics Optimization by Fully Convolutional Neural NetworksIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2018
- Evaluation of quality of sound source separation algorithms: Human perception vs quantitative metrics2016
- Complex Ratio Masking for Monaural Speech SeparationIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2015
- The voice bank corpus: Design, collection and data analysis of a large regional accent speech database2013