Multilingual Text-To-Speech Training Using Cross Language Voice Conversion And Self-Supervised Learning Of Speech Representations
Authors:
- 23 May 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8017-8021
Abstract
State of the art text-to-speech (TTS) models can generate high fidelity monolingual speech, but it is still challenging to synthesize multilingual speech from the same speaker. One major hurdle is for training data. It’s hard to find speakers who have native proficiency in several languages. One way of mitigating this issue is by generating polyglot corpus through voice conversion. In this paper, we train such multilingual TTS system through a novel cross-lingual voice conversion model trained with speaker-invariant features extracted from a speech representation model which is pre-trained with 53 languages through self-supervised learning [1]. To further improve the speaker identity shift, we also adopt a speaker similarity loss term during training. We then use this model to convert multilingual multi-speaker speech data to the voice of the target speaker. Through augmenting data from 4 other languages, we train a multilingual TTS system for a native monolingual English speaker which speaks 5 languages(English, French, German, Italian and Spanish). Our system achieves improved mean opinion score (MOS) compared with the baseline of multi-speaker system for all languages, specifically: 3.74 vs 3.62 for Spanish, 3.11 vs 2.71 for German, 3.47 vs 2.84 for Italian, and 2.72 vs 2.41 for French.Keywords
This publication has 15 references indexed in Scilit:
- Towards Natural Bilingual and Code-Switched Speech Synthesis Based on Mix of Monolingual Recordings and Cross-Lingual Voice Conversion2020
- TTS Skins: Speaker Conversion via ASR2020
- Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning2019
- Neural Speech Synthesis with Transformer NetworkProceedings of the AAAI Conference on Artificial Intelligence, 2019
- An Approach to Cross-Lingual Voice Conversion2019
- Generalized End-to-End Loss for Speaker Verification2018
- Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions2018
- A light-weight method of building an LSTM-RNN-based bilingual tts system2017
- A Multilingual to Polyglot Speech Synthesizer for Indian Languages Using a Voice-Converted Polyglot Speech CorpusCircuits, Systems, and Signal Processing, 2017
- Yet another algorithm for pitch tracking2002