VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesis
Authors:
- 23 May 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 7237-7241
Abstract
In this paper, we propose VISinger, a complete end-to-end high-quality singing voice synthesis (SVS) system that directly generates singing audio from lyrics and musical score. Our approach is inspired by VITS [1], an end-to-end speech generation model which adopts VAE-based posterior encoder augmented with normalizing flow based prior encoder and adversarial decoder. VISinger follows the main architecture of VITS, but makes substantial improvements to the prior encoder according to the characteristics of singing. First, instead of using phoneme-level mean and variance of acoustic features, we introduce a length regulator and a frame prior network to get the frame-level mean and variance on acoustic features, modeling the rich acoustic variation in singing. Second, we further introduce an F0 predictor to guide the frame prior network, leading to stabler singing performance. Finally, to improve the singing rhythm, we modify the duration predictor to specifically predict the phoneme to note duration ratio, helped with singing note normalization. Experiments on a professional Mandarin singing corpus show that VISinger significantly outperforms FastSpeech+Neural-Vocoder two-stage approach and the oracle VITS; ablation study demonstrates the effectiveness of different contributions.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- N-Singer: A Non-Autoregressive Korean Singing Voice Synthesis System for Pronunciation Enhancement2021
- Glow-WaveGAN: Learning Speech Representations from GAN-Based Variational Auto-Encoder for High Fidelity Flow-Based Speech Synthesis2021
- ByteSing: A Chinese Singing Voice Synthesis System Using Duration Allocated Encoder-Decoder Acoustic Models and WaveRNN Vocoders2021
- DurIAN-SC: Duration Informed Attention Network Based Singing Voice Conversion System2020
- DeepSinger: Singing Voice Synthesis with Data Mined From the Web2020
- Sequence-to-Sequence Singing Synthesis Using the Feed-Forward Transformer2020
- Singing Voice Synthesis Based on Generative Adversarial Networks2019
- Recent Development of the DNN-based Singing Voice Synthesis System — Sinsy2018
- Korean Singing Voice Synthesis Based on an LSTM Recurrent Neural Network2018
- WORLD: A Vocoder-Based High-Quality Speech Synthesis System for Real-Time ApplicationsIEICE Transactions on Information and Systems, 2016