Evaluation Metrics for Generative Models: An Empirical Study
Open Access
- 7 July 2024
- journal article
- research article
- Published by MDPI AG in Machine Learning and Knowledge Extraction
- Vol. 6 (3) , 1531-1544
Abstract
Generative models such as generative adversarial networks, diffusion models, and variational auto-encoders have become prevalent in recent years. While it is true that these models have shown remarkable results, evaluating their performance is challenging. This issue is of vital importance to push research forward and identify meaningful gains from random noise. Currently, heuristic metrics such as the inception score (IS) and Fréchet inception distance (FID) are the most common evaluation metrics, but what they measure is not entirely clear. Additionally, there are questions regarding how meaningful their score actually is. In this work, we propose a novel evaluation protocol for likelihood-based generative models, based on generating a high-quality synthetic dataset on which we can estimate classical metrics for comparison. This new scheme harnesses the advantages of knowing the underlying likelihood values of the data by measuring the divergence between the model-generated data and the synthetic dataset. Our study shows that while FID and IS correlate with several f-divergences, their ranking of close models can vary considerably, making them problematic when used for fine-grained comparison. We further use this experimental setting to study which evaluation metric best correlates with our probabilistic metrics.Keywords
Funding Information
- Israeli Council for Higher Education, Data Science Program (“Audience” project)
This publication has 12 references indexed in Scilit:
- Knowing Knowledge: Epistemological Study of Knowledge in TransformersApplied Sciences, 2023
- TREND: Truncated Generalized Normal Density Estimation of Inception Embeddings for GAN Evaluation2022
- Pros and cons of GAN evaluation measures: New developmentsComputer Vision and Image Understanding, 2022
- Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive ModelsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2021
- Effectively Unbiased FID and Inception Score and Where to Find Them2020
- Training Discriminative Models to Evaluate Generative Ones2019
- FairGAN: Fairness-aware Generative Adversarial Networks2018
- How Good Is My GAN?2018
- ImageNet: A large-scale hierarchical image database2009
- A NEW MEASURE OF RANK CORRELATIONBiometrika, 1938