CNN-Based Metrics for Performance Evaluation of Generative Adversarial Networks
- 15 May 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Artificial Intelligence
- Vol. 5 (10) , 5040-5049
Abstract
In this work, we propose two Convolutional Neural Network (CNN) based metrics, Classification Score (CS) and Distribution Score (DS), for performance evaluation of Generative Adversarial Networks (GANs). Though GAN-generated images can be evaluated through manual assessment of visual fidelity, it is prolonged, subjective, challenging, tiresome, and can be misleading. Existing quantitative methods are biased towards memory GAN and fail to detect over-fitting. CS and DS allow us to experimentally prove that training of GANs is actually guided by the data set, that it improves with every epoch and gets closer to following the distribution of the data set. Both methods are based on GAN-generated image classification by CNN. CS is the root mean square (RMS) value of three different classification techniques, Direct Classification (DC), Indirect Classification (IC), and Blind Classification (BC). It exhibits the degree to which GAN can learn the features and generate fake images similar to real data sets. DS shows the contrast between the mean distribution of GAN-generated data and the real data. It indicates the extent to which GANs can create synthetic images with similar distribution to real data sets. We evaluated CS and DS metrics for different variants of GANs and compared their performances with existing metrics. Results show that CS and DS can evaluate the different variants of GANs quantitatively and qualitatively while detecting over-fitting and mode collapse.Keywords
This publication has 14 references indexed in Scilit:
- Pros and cons of GAN evaluation measuresComputer Vision and Image Understanding, 2019
- Generative Adversarial Networks: An OverviewIEEE Signal Processing Magazine, 2018
- Recurrent Topic-Transition GAN for Visual Paragraph Generation2017
- Show, Adapt and Tell: Adversarial Training of Cross-Domain Image Captioner2017
- Learning to generate images with perceptual similarity metrics2017
- A-Fast-RCNN: Hard Positive Generation via Adversary for Object Detection2017
- Semantic Image Inpainting with Deep Generative Models2017
- Learning from Simulated and Unsupervised Images through Adversarial Training2017
- Image-to-Image Translation with Conditional Adversarial Networks2017
- Gist of the Scene2005