Acoustic BPE for Speech Generation with Discrete Tokens
Preprint
Abstract
Discrete audio tokens derived from self-supervised learning models have gained widespread usage in speech generation. However, current practice of directly utilizing audio tokens poses challenges for sequence modeling due to the length of the token sequence. Additionally, this approach places the burden on the model to establish correlations between tokens, further complicating the modeling process. To address this issue, we propose acoustic BPE which encodes frequent audio token patterns by utilizing byte-pair encoding. Acoustic BPE effectively reduces the sequence length and leverages the prior morphological information present in token sequence, which alleviates the modeling challenges of token correlation. Through comprehensive investigations on a speech language model trained with acoustic BPE, we confirm the notable advantages it offers, including faster inference and improved syntax capturing capabilities. In addition, we propose a novel rescore method to select the optimal synthetic speech among multiple candidates generated by rich-diversity TTS system. Experiments prove that rescore selection aligns closely with human preference, which highlights acoustic BPE's potential to other speech generation tasks.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and VocodingProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal SupervisionTransactions of the Association for Computational Linguistics, 2023
- Speaker Adaptive Text-to-Speech With Timbre-Normalized Vector-Quantized FeatureIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023
- AudioLM: A Language Modeling Approach to Audio GenerationIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023
- Multi-Speaker Multi-Lingual VQTTS System for LIMMITS 2023 Challenge2023
- Speechlmscore: Evaluating Speech Generation Using Speech Language Model2023
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech ProcessingIEEE Journal of Selected Topics in Signal Processing, 2022
- w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training2021
- Libri-Light: A Benchmark for ASR with Limited or No Supervision2020
- Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions2018