FPO: Fine-Grained Preference Optimization Improves Zero-Shot Text-to-Speech
- 26 December 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 557-566
Abstract
Integrating reinforcement learning to align generated speech with human preferences has proven effective in improving the robustness of language model-based text-to-speech (TTS) systems. Current approaches primarily rely on preference data annotated at the utterance level. However, frequent issues affecting the listening experience, such as unnatural prosody, mispronunciations, or semantic inconsistencies, often arise only in specific segments of audio, while other segments may be well-generated and require no correction. This mismatch between coarse-grained supervision and fine-grained quality variation limits the effectiveness of preference-based optimization. In this study, we propose a fine-grained preference optimization approach (FPO) to enhance the robustness of TTS systems. FPO shifts the optimization paradigm from global utterance-level tuning to targeted local refinement, focusing on addressing localized issues in generated samples rather than uniformly optimizing the entire utterance. We begin by analyzing the types of common generation issues and categorizing them into temporal modeling errors and semantic-phonetic alignment errors, which frequently degrade intelligibility and naturalness. To tackle these problems, we introduce a selective training loss strategy that leverages fine-grained labels for each issue type, allowing the model to focus on learning signals where they are most needed. Experimental results demonstrate that FPO substantially improves the robustness of zero-shot TTS systems by effectively correcting problematic regions in the output. This leads to a significant reduction in the bad case ratio, improved intelligibility, and overall perceptual quality. Moreover, FPO exhibits strong data efficiency, achieving comparable or superior performance to baseline methods while requiring notably fewer training samples. Audio samples can be found at https://yaoxunji.github.io/fpo/.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Preference Alignment Improves Language Model-Based TTS2025
- Neural Codec Language Models are Zero-Shot Text to Speech SynthesizersIEEE Transactions on Audio, Speech and Language Processing, 2025
- VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild2024
- Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal SupervisionTransactions of the Association for Computational Linguistics, 2023
- AudioLM: A Language Modeling Approach to Audio GenerationIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023
- SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities2023
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech ProcessingIEEE Journal of Selected Topics in Signal Processing, 2022
- w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training2021
- SoundStream: An End-to-End Neural Audio CodecIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden UnitsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021