Zero-Shot Voice Conversion via Content-Aware Timbre Ensemble and Conditional Flow Matching
- 7 October 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 32 (10709908) , 4199-4203
Abstract
Despite recent advances in zero-shot voice conversion (VC), achieving speaker similarity and naturalness comparable to ground-truth recordings remains a significant challenge. In this letter, we propose CTEFM-VC, a zero-shot VC framework that integrates Content-aware Timbre Ensemble modeling with conditional Flow Matching. Specifically, CTEFM-VC decouples utterances into content and timbre representations and leverages a conditional flow matching model to reconstruct the Mel-spectrogram of the source speech. To enhance its timbre modeling capability and naturalness of generated speech, we first introduce a context-aware timbre ensemble modeling approach that adaptively integrates diverse speaker verification embeddings and enables the effective utilization of source content and target timbre elements through a cross-attention module. Furthermore, a structural similarity-based timbre loss is presented to jointly train CTEFM-VC end-to-end. Experiments show that CTEFM-VC consistently achieves the best performance in all metrics assessing speaker similarity, speech naturalness, and intelligibility, significantly outperforming state-of-the-art zero-shot VC systems.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- SEF-VC: Speaker Embedding Free Zero-Shot Voice Conversion with Cross Attention2024
- GEmo-CLAP: Gender-Attribute-Enhanced Contrastive Language-Audio Pretraining for Accurate Speech Emotion Recognition2024
- GPT understands, tooAI Open, 2024
- ACE-VC: Adaptive and Controllable Voice Conversion Using Explicitly Disentangled Self-Supervised Speech Representations2023
- Disentangling Content and Fine-Grained Prosody Information Via Hybrid ASR Bottleneck Features for Voice Conversion2022
- Training Robust Zero-Shot Voice Conversion Models with Self-Supervised Features2022
- Emotional voice conversion: Theory, databases and ESDSpeech Communication, 2022
- Zero-Shot Voice Conversion with Adjusted Speaker Embeddings and Simple Acoustic Features2021
- Evaluating Voice Conversion-Based Privacy Protection against Informed Attackers2020
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015