Explainable AI in the military domain
Open Access
- 16 April 2024
- journal article
- research article
- Published by Springer Nature in Ethics and Information Technology
- Vol. 26 (2) , 1-13
Abstract
Artificial intelligence (AI) has become nearly ubiquitous in modern society, from components of mobile applications to medical support systems, and everything in between. In societally impactful systems imbued with AI, there has been increasing concern related to opaque AI, that is, artificial intelligence where it is unclear how or why certain decisions are reached. This has led to a recent boom in research on “explainable AI” (XAI), or approaches to making AI more explainable and understandable to human users. In the military domain, numerous bodies have argued that autonomous and AI-enabled weapon systems ought not incorporate unexplainable AI, with the International Committee of the Red Cross and the United States Department of Defense both explicitly including explainability as a relevant factor in the development and use of such systems. In this article, I present a cautiously critical assessment of this view, arguing that explainability will be irrelevant for many current and near-future autonomous systems in the military (which do not incorporate any AI), that it will be trivially incorporated into most military systems which do possess AI (as these generally possess simpler AI systems), and that for those systems with genuinely opaque AI, explainability will prove to be of more limited value than one might imagine. In particular, I argue that explainability, while indeed a virtue in design, is a virtue aimed primarily at designers and troubleshooters of AI-enabled systems, but is far less relevant for users and handlers actually deploying these systems. I further argue that human–machine teaming is a far more important element of responsibly using AI for military purposes, adding that explainability may undermine efforts to improve human–machine teamings by creating aprima faciesense that the AI, due to its explainability, may be utilized with little (or less) potential for mistakes. I conclude by clarifying that the arguments are not against XAI in the military, but are instead intended as a caution against over-inflating the value of XAI in this domain, or ignoring the limitations and potential pitfalls of this approach.Keywords
Funding Information
- Grantová Agentura České Republiky (24-12638I)
This publication has 39 references indexed in Scilit:
- Autonomous weapon systems and responsibility gaps: a taxonomyEthics and Information Technology, 2023
- Legal reviews of in situ learning in autonomous weaponsEthics and Information Technology, 2023
- AI and the expert; a blueprint for the ethical use of opaque AIAI & SOCIETY, 2022
- A survey on XAI and natural language explanationsInformation Processing & Management, 2022
- A Comparative Analysis of the Definitions of Autonomous Weapons SystemsScience and Engineering Ethics, 2022
- What do we want from Explainable Artificial Intelligence (XAI)? – A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI researchArtificial Intelligence, 2021
- Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AIInformation Fusion, 2020
- Meaningful human control as reason-responsiveness: the case of dual-mode vehiclesEthics and Information Technology, 2019
- Explanation in artificial intelligence: Insights from the social sciencesArtificial Intelligence, 2019
- Information Overload: A Decision Making Perspective2001