Beyond Prediction Similarity: ShapGAP for Evaluating Faithful Surrogate Models in XAI
Open Access
- 30 October 2023
- book chapter
- Published by Springer Nature
- p. 160-173
Abstract
The growing importance of Explainable Artificial Intelligence (XAI) has highlighted the need to understand the decision-making processes of black-box models. Surrogation, emulating a black-box model (BB) with a white-box model (WB), is crucial in applications where BBs are unavailable due to security or practical concerns. Traditional fidelity measures only evaluate the similarity of the final predictions, which can lead to a significant limitation: considering a WB faithful even when it has the same prediction as the BB but with a completely different rationale. Addressing this limitation is crucial to develop Trustworthy AI practical applications beyond XAI. To address this issue, we introduce ShapGAP, a novel metric that assesses the faithfulness of surrogate models by comparing their reasoning paths, using SHAP explanations as a proxy. We validate the effectiveness of ShapGAP by applying it to real-world datasets from healthcare and finance domains, comparing its performance against traditional fidelity measures. Our results show that ShapGAP enables better understanding and trust in XAI systems, revealing the potential dangers of relying on models with high task accuracy but unfaithful explanations. ShapGAP serves as a valuable tool for identifying faithful surrogate models, paving the way for more reliable and Trustworthy AI applications.Keywords
This publication has 17 references indexed in Scilit:
- Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AIInformation Fusion, 2020
- Stop explaining black box machine learning models for high stakes decisions and use interpretable models insteadNature Machine Intelligence, 2019
- Distill-and-Compare2018
- AI4People—An Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and RecommendationsMinds and Machines, 2018
- The Intuitive Appeal of Explainable MachinesSSRN Electronic Journal, 2018
- Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPRSSRN Electronic Journal, 2017
- The ethics of algorithms: Mapping the debateBig Data & Society, 2016
- "Why Should I Trust You?"2016
- Nuclear feature extraction for breast tumor diagnosis1993
- 17. A Value for n-Person Games1953