How Algorithms Discriminate Based on Data They Lack: Challenges, Solutions, and Policy Implications
Authors:
Open Access
- 1 March 2018
- journal article
- research article
- Published by The Pennsylvania State University Press in Journal of Information Policy
- Vol. 8, 78-115
Abstract
Organizations often employ data-driven models to inform decisions that can have a significant impact on people's lives (e.g., university admissions, hiring). In order to protect people's privacy and prevent discrimination, these decision-makers may choose to delete or avoid collecting social category data, like sex and race. In this article, we argue that such censoring can exacerbate discrimination by making biases more difficult to detect. We begin by detailing how computerized decisions can lead to biases in the absence of social category data and in some contexts, may even sustain biases that arise by random chance. We then show how proactively using social category data can help illuminate and combat discriminatory practices, using cases from education and employment that lead to strategies for detecting and preventing discrimination. We conclude that discrimination can occur in any sociotechnical system in which someone decides to use an algorithmic process to inform decision-making, and we offer a set of broader implications for researchers and policymakers.Keywords
This publication has 58 references indexed in Scilit:
- Implementing Anti-Discrimination Policies in Statistical Profiling ModelsAmerican Economic Journal: Economic Policy, 2011
- Accuracy and Inaccuracy in Teachers’ Perceptions of Young Children’s Cognitive AbilitiesAmerican Educational Research Journal, 2011
- THEORIZING DIGITAL AND URBAN INEQUALITIESInformation, Communication & Society, 2010
- Engaging rational discrimination: exploring reasons for placing regulatory constraints on decision support systemsEthics and Information Technology, 2009
- The Time- and Context-Contingent Nature of Intersectionality and Interlocking OppressionsAffilia, 2009
- Information and the Change in the Paradigm in EconomicsAmerican Economic Review, 2002
- Signaling in Retrospect and the Informational Structure of MarketsAmerican Economic Review, 2002
- The Enforcement of Equal Opportunity Laws Under Imperfect Information: Affirmative Action and AlternativesThe Quarterly Journal of Economics, 1991
- Discrimination, Nepotism, and Long-Run Wage DifferentialsThe Quarterly Journal of Economics, 1982
- Job Market SignalingThe Quarterly Journal of Economics, 1973