- 6 April 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 34 (11) , 9439-9450
Abstract
In this article, we propose a novel loss function for training generative adversarial networks (GANs) aiming toward deeper theoretical understanding as well as improved stability and performance for the underlying optimization problem. The new loss function is based on cumulant generating functions (CGFs) giving rise to Cumulant GAN. Relying on a recently derived variational formula, we show that the corresponding optimization problem is equivalent to Rényi divergence minimization, thus offering a (partially) unified perspective of GAN losses: the Rényi family encompasses Kullback-Leibler divergence (KLD), reverse KLD, Hellinger distance, and χ²-divergence. Wasserstein GAN is also a member of cumulant GAN. In terms of stability, we rigorously prove the linear convergence of cumulant GAN to the Nash equilibrium for a linear discriminator, Gaussian distributions, and the standard gradient descent ascent algorithm. Finally, we experimentally demonstrate that image generation is more robust relative to Wasserstein GAN and it is substantially improved in terms of both inception score (IS) and Fréchet inception distance (FID) when both weaker and stronger discriminators are considered.Keywords
This publication has 23 references indexed in Scilit:
- Statistical Parametric Speech Synthesis Incorporating Generative Adversarial NetworksIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2017
- Least Squares Generative Adversarial Networks2017
- Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks2017
- SEGAN: Speech Enhancement Generative Adversarial Network2017
- Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network2017
- Adversarial Discriminative Domain Adaptation2017
- Neural Information Processing2017
- Convergence Rates in Forward--Backward SplittingSIAM Journal on Optimization, 1997
- Moments and cumulants of the multivariate normal distributionStochastic Analysis and Applications, 1988
- Asymptotic evaluation of certain markov process expectations for large time. IVCommunications on Pure and Applied Mathematics, 1983