SEF-VC: Speaker Embedding Free Zero-Shot Voice Conversion with Cross Attention
Preprint
Abstract
Zero-shot voice conversion (VC) aims to transfer the source speaker timbre to arbitrary unseen target speaker timbre, while keeping the linguistic content unchanged. Although the voice of generated speech can be controlled by providing the speaker embedding of the target speaker, the speaker similarity still lags behind the ground truth recordings. In this paper, we propose SEF-VC, a speaker embedding free voice conversion model, which is designed to learn and incorporate speaker timbre from reference speech via a powerful position-agnostic cross-attention mechanism, and then reconstruct waveform from HuBERT semantic tokens in a non-autoregressive manner. The concise design of SEF-VC enhances its training stability and voice conversion performance. Objective and subjective evaluations demonstrate the superiority of SEF-VC to generate high-quality speech with better similarity to target reference than strong zero-shot VC baselines, even for very short reference speeches.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and VocodingProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Speaker Adaptive Text-to-Speech With Timbre-Normalized Vector-Quantized FeatureIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023
- LM-VC: Zero-Shot Voice Conversion via Speech Generation Based on Language ModelsIEEE Signal Processing Letters, 2023
- AudioLM: A Language Modeling Approach to Audio GenerationIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden UnitsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech2019
- One-Shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization2019
- Voice conversion from non-parallel corpora using variational auto-encoder2016