Algorithmic Decision-Making and the Control Problem
Open Access
- 11 December 2019
- journal article
- research article
- Published by Springer Nature in Minds and Machines
- Vol. 29 (4) , 555-578
Abstract
The danger of human operators devolving responsibility to machines and failing to detect cases where they fail has been recognised for many years by industrial psychologists and engineers studying the human operators of complex machines. We call it “the control problem”, understood as the tendency of the human within a human–machine control loop to become complacent, over-reliant or unduly diffident when faced with the outputs of a reliable autonomous system. While the control problem has been investigated for some time, up to this point its manifestation in machine learning contexts has not received serious attention. This paper aims to fill that gap. We argue that, except in certain special circumstances, algorithmic decision tools should not be used in high-stakes or safety-critical decisions unless the systems concerned are significantly “better than human” in the relevant domain or subdomain of decision-making. More concretely, we recommend three strategies to address the control problem, the most promising of which involves a complementary (and potentially dynamic) coupling between highly proficient algorithmic tools and human agents working alongside one another. We also identify six key principles which all such human–machine systems should reflect in their design. These can serve as a framework both for assessing the viability of any such human–machine system as well as guiding the design and implementation of such systems generally.Keywords
Funding Information
- New Zealand Law Foundation (2016/ILP/10)
This publication has 27 references indexed in Scilit:
- Distributed situation awarenessTheoretical Issues in Ergonomics Science, 2015
- Extraneous factors in judicial decisionsProceedings of the National Academy of Sciences, 2011
- Complacency and Bias in Human Use of Automation: An Attentional IntegrationHuman Factors: The Journal of the Human Factors and Ergonomics Society, 2010
- Effective forecasting and judgmental adjustments: an empirical evaluation and strategies for improvement in supply-chain planningInternational Journal of Forecasting, 2009
- Cognitive Elements of Human Decision Making2008
- Accountability and automation biasInternational Journal of Human-Computer Studies, 2000
- From fly-by-wire to drive-by-wire: Safety implications of automation in vehiclesSafety Science, 1996
- Ironies of automationAutomatica, 1983
- Human-Computer Interaction in the Control of Dynamic SystemsACM Computing Surveys, 1981
- Flight-deck automation: promises and problemsErgonomics, 1980