SC-CNN: Effective Speaker Conditioning Method for Zero-Shot Multi-Speaker Text-to-Speech Systems
- 18 May 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 30 (10709908) , 593-597
Abstract
This letter proposes an effective speaker-conditioning method that is applicable to zero-shot multi-speaker text-to-speech (ZSM-TTS) systems. Based on the inductive bias in the speech generation task, in which local context information in text/phoneme sequences heavily affect the speaker characteristics of the output speech, we propose a Speaker-Conditional Convolutional Neural Network (SC-CNN) for the ZSM-TTS task. SC-CNN first predicts convolutional kernels from each learned speaker embedding, then applies 1-D convolutions to phoneme sequences with the predicted kernels. It utilizes the aforementioned inductive bias and effectively models the characteristic of speech by providing the speaker-specific local context in phonetic domain. We also build both FastSpeech2 and VITS-based ZSM-TTS systems to verify its superiority over conventional speaker conditioning methods. The results confirm that the models with SC-CNN outperform the recent ZSM-TTS models in terms of both subjective and objective measurements.Keywords
This publication has 14 references indexed in Scilit:
- MultiSpeech: Multi-Speaker Text to Speech with Transformer2020
- ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification2020
- Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding2020
- Conformer: Convolution-augmented Transformer for Speech Recognition2020
- BOFFIN TTS: Few-Shot Speaker Adaptation by Bayesian Optimization2020
- Mellotron: Multispeaker Expressive Voice Synthesis by Conditioning on Rhythm, Pitch and Global Style Tokens2020
- LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech2019
- Neural Speech Synthesis with Transformer NetworkProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions2018
- Xception: Deep Learning with Depthwise Separable Convolutions2017