Vevo2: A Unified and Controllable Framework for Speech and Singing Voice Generation
- 9 March 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 2262-2275
Abstract
Controllable human voice generation, particularly for expressive domains like singing, remains a significant challenge. This paper introduces ${\bm {Vevo2}}$, a unified framework for controllable speech and singing voice generation. To tackle issues like the scarcity of annotated singing data and to enable flexible controllability, Vevo2 introduces two audio tokenizers: (1) a unified music-notation-free prosody tokenizer that captures prosody and melody from speech, singing, and even instrumental sounds, and (2) a unified content-style tokenizer that encodes linguistic content, prosody, and style for both speech and singing, while enabling timbre disentanglement. Vevo2 consists of an auto-regressive content-style modeling stage, which aims to enable controllability over text, prosody, and style, as well as a flow-matching acoustic modeling stage that allows for timbre control. Particularly, during the speech-singing joint training of the AR model, we propose both explicit and implicit prosody learning strategies to bridge speech and singing voice. Moreover, to further enhance the Vevo2's ability to follow text and prosody, we design a multi-objective post-training task that integrates both intelligibility and prosody similarity alignment. Experimental results show that the unified modeling in Vevo2 brings mutual benefits to both speech and singing voice generation. Additionally, Vevo2's effectiveness across a wide range of synthesis, conversion, and editing tasks for both speech and singing further demonstrates its strong generalization ability and versatility. Audio samples are are available at https://versasinger.github.io/.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62376237)
- 2023 Shenzhen Stability Science Program
- Shenzhen Research Institute of Big Data (T00120230002)
- Program for Guangdong Introducing Innovative and Enterpreneurial Teams (2023ZT10X044)
This publication has 58 references indexed in Scilit:
- Neural Concatenative Singing Voice Conversion: Rethinking Concatenation-Based Approach for One-Shot Singing Voice Conversion2024
- Humtrans: A Novel Open-Source Dataset for Humming Melody Transcription and Beyond2024
- Vits-Based Singing Voice Conversion Leveraging Whisper and Multi-Scale F0 Modeling2023
- The Singing Voice Conversion Challenge 20232023
- Neural Text-to-Speech Synthesis2023
- Deep Performer: Score-to-Audio Music Performance Synthesis2022
- VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesis2022
- Emotional voice conversion: Theory, databases and ESDSpeech Communication, 2022
- w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training2021
- Libri-Light: A Benchmark for ASR with Limited or No Supervision2020