Counterfactual Edits for Generative Evaluation
Preprint
Abstract
Evaluation of generative models has been an underrepresented field despite the surge of generative architectures. Most recent models are evaluated upon rather obsolete metrics which suffer from robustness issues, while being unable to assess more aspects of visual quality, such as compositionality and logic of synthesis. At the same time, the explainability of generative models remains a limited, though important, research direction with several current attempts requiring access to the inner functionalities of generative models. Contrary to prior literature, we view generative models as a black box, and we propose a framework for the evaluation and explanation of synthesized results based on concepts instead of pixels. Our framework exploits knowledge-based counterfactual edits that underline which objects or attributes should be inserted, removed, or replaced from generated images to approach their ground truth conditioning. Moreover, global explanations produced by accumulating local edits can also reveal what concepts a model cannot generate in total. The application of our framework on various models designed for the challenging tasks of Story Visualization and Scene Synthesis verifies the power of our approach in the model-agnostic setting.Keywords
All Related Versions
This publication has 29 references indexed in Scilit:
- Limited Discriminator GAN using explainable AI model for overfitting problemICT Express, 2022
- Pros and cons of GAN evaluation measures: New developmentsComputer Vision and Image Understanding, 2022
- Evaluation Metrics for Conditional Image GenerationInternational Journal of Computer Vision, 2021
- Learning Visual Commonsense for Robust Scene Graph Generation2020
- Improved-StoryGAN for sequential images visualizationJournal of Visual Communication and Image Representation, 2020
- DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis2019
- Semantic Image Synthesis With Spatially-Adaptive Normalization2019
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks2018
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- Microsoft COCO: Common Objects in Context2014