Collective Learning Mechanism based Optimal Transport Generative Adversarial Network for Non-parallel Voice Conversion
Preprint
Abstract
After demonstrating significant success in image synthesis, Generative Adversarial Network (GAN) models have likewise made significant progress in the field of speech synthesis, leveraging their capacity to adapt the precise distribution of target data through adversarial learning processes. Notably, in the realm of State-Of-The-Art (SOTA) GAN-based Voice Conversion (VC) models, there exists a substantial disparity in naturalness between real and GAN-generated speech samples. Furthermore, while many GAN models currently operate on a single generator discriminator learning approach, optimizing target data distribution is more effectively achievable through a single generator multi-discriminator learning scheme. Hence, this study introduces a novel GAN model named Collective Learning Mechanism-based Optimal Transport GAN (CLOT-GAN) model, incorporating multiple discriminators, including the Deep Convolutional Neural Network (DCNN) model, Vision Transformer (ViT), and conformer. The objective of integrating various discriminators lies in their ability to comprehend the formant distribution of mel-spectrograms, facilitated by a collective learning mechanism. Simultaneously, the inclusion of Optimal Transport (OT) loss aims to precisely bridge the gap between the source and target data distribution, employing the principles of OT theory. The experimental validation on VCC 2018, VCTK, and CMU-Arctic datasets confirms that the CLOT-GAN-VC model outperforms existing VC models in objective and subjective assessments.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- GLGAN-VC: A Guided Loss-Based Generative Adversarial Network for Many-to-Many Voice ConversionIEEE Transactions on Neural Networks and Learning Systems, 2023
- Towards solving NLP tasks with optimal transport lossJournal of King Saud University Computer and Information Sciences, 2022
- An Adaptive-Learning-Based Generative Adversarial Network for One-to-One Voice ConversionIEEE Transactions on Artificial Intelligence, 2022
- Flattening Layer Pruning in Convolutional Neural NetworksSymmetry, 2021
- An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep LearningIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020
- Voice Conversion for Whispered Speech SynthesisIEEE Signal Processing Letters, 2019
- Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based LocalizationInternational Journal of Computer Vision, 2019
- The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methods2018
- Restricted Boltzmann Machine-Based Voice Conversion for Nonparallel CorpusIEEE Signal Processing Letters, 2017
- Mixture of Factor Analyzers Using Priors From Non-Parallel Speech for Voice ConversionIEEE Signal Processing Letters, 2012