The Role of Explainability in Assuring Safety of Machine Learning in Healthcare
Open Access
- 5 May 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Emerging Topics in Computing
- Vol. 10 (4) , 1746-1760
Abstract
Established approaches to assuring safety-critical systems and software are difficult to apply to systems employing ML where there is no clear, pre-defined specification against which to assess validity. This problem is exacerbated by the opaque nature of ML where the learnt model is not amenable to human scrutiny. XAI methods have been proposed to tackle this issue by producing human-interpretable representations of ML models which can help users to gain confidence and build trust in the ML system. However, little work explicitly investigates the role of explainability for safety assurance in the context of ML development. This paper identifies ways in which XAI methods can contribute to safety assurance of ML-based systems. It then uses a concrete ML-based clinical decision support system, concerning weaning of patients from mechanical ventilation, to demonstrate how XAI methods can be employed to produce evidence to support safety assurance. The results are also represented in a safety argument to show where, and in what way, XAI methods can contribute to a safety case. Overall, we conclude that XAI methods have a valuable role in safety assurance of ML-based systems in healthcare but that they are not sufficient in themselves to assure safety.Keywords
All Related Versions
Funding Information
- Bradford Teaching Hospitals NHS Foundation Trust
- University of York
This publication has 27 references indexed in Scilit:
- Feature VisualizationDistill, 2017
- Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations2017
- Towards Evaluating the Robustness of Neural Networks2017
- A Harmonized Data Quality Assessment Terminology and Framework for the Secondary Use of Electronic Health Record DataeGEMs (Generating Evidence & Methods to improve patient outcomes), 2016
- "Why Should I Trust You?"2016
- On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance PropagationPLOS ONE, 2015
- Peeking Inside the Black Box: Visualizing Statistical Learning With Plots of Individual Conditional ExpectationJournal of Computational and Graphical Statistics, 2015
- The difficult-to-wean patientExpert Review of Respiratory Medicine, 2010
- Logistic Regression: Why We Cannot Do What We Think We Can Do, and What We Can Do About ItEuropean Sociological Review, 2009
- Evaluation of simple criteria to predict successful weaning from mechanical ventilation in intensive care patientsBritish Journal of Anaesthesia, 2004