Black-Box Testing and Auditing of Bias in ADM Systems
Open Access
- 25 May 2024
- journal article
- research article
- Published by Springer Nature in Minds and Machines
- Vol. 34 (2) , 1-31
Abstract
For years, the number of opaque algorithmic decision-making systems (ADM systems) with a large impact on society has been increasing: e.g., systems that compute decisions about future recidivism of criminals, credit worthiness, or the many small decision computing systems within social networks that create rankings, provide recommendations, or filter content. Concerns that such a system makes biased decisions can be difficult to investigate: be it by people affected, NGOs, stakeholders, governmental testing and auditing authorities, or other external parties. Scientific testing and auditing literature rarely focuses on the specific needs for such investigations and suffers from ambiguous terminologies. With this paper, we aim to support this investigation process by collecting, explaining, and categorizing methods of testing for bias, which are applicable to black-box systems, given that inputs and respective outputs can be observed. For this purpose, we provide a taxonomy that can be used to select suitable test methods adapted to the respective situation. This taxonomy takes multiple aspects into account, for example the effort to implement a given test method, its technical requirement (such as the need of ground truth) and social constraints of the investigation, e.g., the protection of business secrets. Furthermore, we analyze which test method can be used in the context of which black box audit concept. It turns out that various factors, such as the type of black box audit or the lack of an oracle, may limit the selection of applicable tests. With the help of this paper, people or organizations who want to test an ADM system for bias can identify which test methods and auditing concepts are applicable and what implications they entail.Keywords
Funding Information
- Volkswagen Foundation
- Bundesministerium für Forschung, Technologie und Raumfahrt (DKI.00.0002 3.20)
- Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau
This publication has 77 references indexed in Scilit:
- Algorithm Assurance: Auditing Applications of Artificial Intelligence2022
- Fairness by awareness? On the inclusion of protected features in algorithmic decisionsComputer Law & Security Review, 2022
- Legal perspective on possible fairness measures – A legal discussion using the example of hiring decisionsComputer Law & Security Review, 2021
- Why Do We Need to Be Bots? What Prevents Society from Detecting Biases in Recommendation Systems2020
- Quantitative analysis of automatic performance evaluation systems based on the h-indexScientometrics, 2020
- On Testing Data-Intensive Software Systems2019
- Dissecting the Algorithmic Leviathan: On the Socio-Political Anatomy of Algorithmic GovernancePhilosophy & Technology, 2019
- An Introduction to Audit Studies in the Social Sciences2018
- Online Controlled Experiments and A/B Testing2017
- Using sensitive personal data may be necessary for avoiding discrimination in data-driven decision modelsArtificial Intelligence and Law, 2016