Evaluating the Stability of Semantic Concept Representations in CNNs for Robust Explainability
Open Access
- 21 October 2023
- book chapter
- Published by Springer Nature
- p. 499-524
Abstract
Analysis of how semantic concepts are represented within Convolutional Neural Networks (CNNs) is a widely used approach in Explainable Artificial Intelligence (XAI) for interpreting CNNs. A motivation is the need for transparency in safety-critical AI-based systems, as mandated in various domains like automated driving. However, to use the concept representations for safety-relevant purposes, like inspection or error retrieval, these must be of high quality and, in particular, stable. This paper focuses on two stability goals when working with concept representations in computer vision CNNs: stability of concept retrieval and of concept attribution. The guiding use-case is a post-hoc explainability framework for object detection (OD) CNNs, towards which existing concept analysis (CA) methods are successfully adapted. To address concept retrieval stability, we propose a novel metric that considers both concept separation and consistency, and is agnostic to layer and concept representation dimensionality. We then investigate impacts of concept abstraction level, number of concept training samples, CNN size, and concept representation dimensionality on stability. For concept attribution stability we explore the effect of gradient instability on gradient-based explainability methods. The results on various CNNs for classification and object detection yield the main findings that (1) the stability of concept retrieval can be enhanced through dimensionality reduction via data aggregation, and (2) in shallow layers where gradient instability is more pronounced, gradient smoothing techniques are advised. Finally, our approach provides valuable insights into selecting the appropriate layer and concept representation dimensionality, paving the way towards CA in safety-critical XAI applications.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- Regression Concept Vectors for Bidirectional Explanations in Histopathology2018
- Interpretable Convolutional Neural Networks2018
- Net2Vec: Quantifying and Explaining How Concepts are Encoded by Filters in Deep Neural Networks2018
- Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks2017
- European Union Regulations on Algorithmic Decision Making and a “Right to Explanation”AI Magazine, 2017
- Network Dissection: Quantifying Interpretability of Deep Visual Representations2017
- SSD: Single Shot MultiBox Detector2016
- Deep Residual Learning for Image Recognition2016
- Microsoft COCO: Common Objects in Context2014
- ImageNet: A large-scale hierarchical image database2009