Counterfactual Image Editing
Preprint
Abstract
Counterfactual image editing is an important task in generative AI, which asks how an image would look if certain features were different. The current literature on the topic focuses primarily on changing individual features while remaining silent about the causal relationships between these features, as present in the real world. In this paper, we formalize the counterfactual image editing task using formal language, modeling the causal relationships between latent generative factors and images through a special type of model called augmented structural causal models (ASCMs). Second, we show two fundamental impossibility results: (1) counterfactual editing is impossible from i.i.d. image samples and their corresponding labels alone; (2) even when the causal relationships between the latent generative factors and images are available, no guarantees regarding the output of the model can be provided. Third, we propose a relaxation for this challenging problem by approximating non-identifiable counterfactual distributions with a new family of counterfactual-consistent estimators. This family exhibits the desirable property of preserving features that the user cares about across both factual and counterfactual worlds. Finally, we develop an efficient algorithm to generate counterfactual images by leveraging neural causal models.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A ReviewACM Computing Surveys, 2024
- Causal Fairness Analysis: A Causal Toolkit for Fair Machine LearningFoundations and Trends in Machine Learning, 2024
- Image Super-Resolution Via Iterative RefinementIEEE Transactions on Pattern Analysis and Machine Intelligence, 2022
- Causal Interpretability for Machine Learning - Problems, Methods and EvaluationACM SIGKDD Explorations Newsletter, 2020
- The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]IEEE Signal Processing Magazine, 2012
- Causality: Models, Reasoning and InferencePhilosophical Review, 2001
- Nonlinear independent component analysis: Existence and uniqueness resultsNeural Networks, 1999
- Counterfactual Probabilities: Computational Methods, Bounds and Applications1994