$α$-GAN by Rényi Cross Entropy
Preprint
Abstract
This paper proposes $α$-GAN, a generative adversarial network using Rényi measures. The value function is formulated, by Rényi cross entropy, as an expected certainty measure incurred by the discriminator's soft decision as to where the sample is from, true population or the generator. The discriminator tries to maximize the Rényi certainty about sample source, while the generator wants to reduce it by injecting fake samples. This forms a min-max problem with the solution parameterized by the Rényi order $α$. This $α$-GAN reduces to vanilla GAN at $α= 1$, where the value function is exactly the binary cross entropy. The optimization of $α$-GAN is over probability (vector) space. It is shown that the gradient is exponentially enlarged when Rényi order is in the range $α\in (0,1)$. This makes convergence faster, which is verified by experimental results. A discussion shows that choosing $α\in (0,1)$ may be able to solve some common problems, e.g., vanishing gradient. A following observation reveals that this range has not been fully explored in the existing Rényi version GANs.
Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Addressing GAN Training Instabilities via Tunable Classification LossesIEEE Journal on Selected Areas in Information Theory, 2024
- A Unifying Generator Loss Function for Generative Adversarial NetworksEntropy, 2024
- A Tunable Loss Function for Robust Classification: Calibration, Landscape, and GeneralizationIEEE Transactions on Information Theory, 2022
- Accelerated gradient algorithm for RBF neural networkNeurocomputing, 2021
- Tunable Measures for Information Leakage and Applications to Privacy-Utility TradeoffsIEEE Transactions on Information Theory, 2019
- Adaptive Restart for Accelerated Gradient SchemesFoundations of Computational Mathematics, 2013
- Estimating Divergence Functionals and the Likelihood Ratio by Convex Risk MinimizationIEEE Transactions on Information Theory, 2010
- On Divergences and Informations in Statistics and Information TheoryIEEE Transactions on Information Theory, 2006
- The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem SolutionsInternational Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 1998