Assessing Algorithmic Fairness with Unobserved Protected Class Using Data Combination
Top Cited Papers
- 1 March 2022
- journal article
- research article
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Management Science
- Vol. 68 (3) , 1959-1981
Abstract
The increasing impact of algorithmic decisions on people’s lives compels us to scrutinize their fairness and, in particular, the disparate impacts that ostensibly color-blind algorithms can have on different groups. Examples include credit decisioning, hiring, advertising, criminal justice, personalized medicine, and targeted policy making, where in some cases legislative or regulatory frameworks for fairness exist and define specific protected classes. In this paper we study a fundamental challenge to assessing disparate impacts in practice: protected class membership is often not observed in the data. This is particularly a problem in lending and healthcare. We consider the use of an auxiliary data set, such as the U.S. census, to construct models that predict the protected class from proxy variables, such as surname and geolocation. We show that even with such data, a variety of common disparity measures are generally unidentifiable, providing a new perspective on the documented biases of popular proxy-based methods. We provide exact characterizations of the tightest possible set of all possible true disparities that are consistent with the data (and possibly additional assumptions). We further provide optimization-based algorithms for computing and visualizing these sets and statistical tools to assess sampling uncertainty. Together, these enable reliable and robust assessments of disparities—an important tool when disparity assessment can have far-reaching policy implications. We demonstrate this in two case studies with real data: mortgage lending and personalized medicine dosing. This paper was accepted by Hamid Nazerzadeh, Management Science Special Section on Data-Driven Prescriptive Analytics.Keywords
This publication has 36 references indexed in Scilit:
- Advancing Health Care Equity through Improved Data CollectionNew England Journal of Medicine, 2011
- Sharp Identification Regions in Models With Convex Moment PredictionsEconometrica, 2011
- Estimation of the Warfarin Dose with Clinical and Pharmacogenetic DataNew England Journal of Medicine, 2009
- Market Structure and Multiple Equilibria in Airline MarketsEconometrica, 2009
- Fast learning rates for plug-in classifiersThe Annals of Statistics, 2007
- Partial identification with missing data: concepts and findingsInternational Journal of Approximate Reasoning, 2005
- Use Of Geocoding In Managed Care Settings To Identify Quality DisparitiesHealth Affairs, 2005
- Ecological Inference for 2 × 2 TablesJournal of the Royal Statistical Society Series A: Statistics in Society, 2004
- Stability Theory for Systems of Inequalities. Part I: Linear SystemsSIAM Journal on Numerical Analysis, 1975
- Programming with linear fractional functionalsNaval Research Logistics Quarterly, 1962