KANGAN-AVSS: Kolmogorov-Arnold Network Based Generative Adversarial Networks for Audio-Visual Speech Synthesis
Authors:
- 6 April 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-5
Abstract
Audio-visual speech synthesis (AVSS) is an emerging research topic in the paradigm of generative AI, aiming to generate realistic and synchronized audio-visual outputs for a target speaker based on input audio from any source speaker, combining Voice Conversion (VC) and Audio-Visual Synthesis (AVS). The VC component transforms the audio characteristics of the source speaker to match the target speaker, while the AVS component generates a video that is synchronized with the converted audio. This paper introduces a novel approach to AVSS by incorporating the Kolmogorov Arnold Network (KAN) into the discriminator of Generative Adversarial Networks (GANs) for both VC and AVS tasks. By integrating KAN into the GAN framework, the proposed approach leverages KAN’s ability to model high-dimensional, non-linear relationships with fewer parameters, enhancing the discriminator’s efficiency and accuracy. This results in more natural synthesis of audio-visual outputs with reduced computational complexity and faster training time. Experimental evaluations on benchmark datasets demonstrate that our KAN-based GAN substantially outperforms existing methods in the quality and naturalness of generated samples. This work advances the state of AVSS by improving the realism of synthesized speech and video, offering a robust and efficient solution for creating synchronized audio-visual output.Keywords
This publication has 31 references indexed in Scilit:
- A Comprehensive Analysis on Features and Performance Evaluation Metrics in Audio-Visual Voice Conversion2024
- Audio-visual speech synthesis using vision transformer–enhanced autoencoders with ensemble of loss functionsApplied Intelligence, 2024
- Region Normalized Capsule Network Based Generative Adversarial Network for Non-parallel Voice Conversion2023
- Rectified Adam Optimizer-Based CNN Model for Speaker Identification2022
- Model complexity of deep learning: a surveyKnowledge and Information Systems, 2021
- Error bounds for deep ReLU networks using the Kolmogorov–Arnold superposition theoremNeural Networks, 2020
- On the Study of Generative Adversarial Networks for Cross-Lingual Voice Conversion2019
- Review of Deep Learning Algorithms and ArchitecturesIEEE Access, 2019
- Mean opinion score (MOS) revisited: methods and applications, limitations and alternativesMultimedia Systems, 2014
- Recurrent neural networks and robust time series predictionIEEE Transactions on Neural Networks, 1994