SEF-VC: Speaker Embedding Free Zero-Shot Voice Conversion with Cross Attention
Authors:
- 14 April 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 12296-12300
Abstract
Zero-shot voice conversion (VC) aims to transfer the source speaker timbre to arbitrary unseen target speaker timbre, while keeping the linguistic content unchanged. Although the voice of generated speech can be controlled by providing the speaker embedding of the target speaker, the speaker similarity still lags behind the ground truth recordings. In this paper, we propose SEF-VC, a speaker embedding free voice conversion model, which is designed to learn and incorporate speaker timbre from reference speech via a powerful position-agnostic cross-attention mechanism, and then reconstruct waveform from HuBERT semantic tokens in a non-autoregressive manner. The concise design of SEF-VC enhances its training stability and voice conversion performance. Objective and subjective evaluations demonstrate the superiority of SEF-VC to generate high-quality speech with better similarity to target reference than strong zero-shot VC baselines, even for very short reference speeches.Keywords
All Related Versions
Funding Information
- Research and Development
This publication has 22 references indexed in Scilit:
- ACE-VC: Adaptive and Controllable Voice Conversion Using Explicitly Disentangled Self-Supervised Speech Representations2023
- Robust Disentangled Variational Speech Representation Learning for Zero-Shot Voice Conversion2022
- DRVC: A Framework of Any-to-Any Voice Conversion with Self-Supervised Learning2022
- DGC-Vector: A New Speaker Embedding for Zero-Shot Voice Conversion2022
- Training Robust Zero-Shot Voice Conversion Models with Self-Supervised Features2022
- Zero-Shot Voice Conversion with Adjusted Speaker Embeddings and Simple Acoustic Features2021
- Any-to-One Sequence-to-Sequence Voice Conversion Using Self-Supervised Discrete Speech Representations2021
- Non-Parallel Voice Conversion Using Variational Autoencoders Conditioned by Phonetic Posteriorgrams and D-Vectors2018
- X-Vectors: Robust DNN Embeddings for Speaker Recognition2018
- Voice conversion from non-parallel corpora using variational auto-encoder2016