Can I Trust My Fairness Metric? Assessing Fairness with Unlabeled Data and Bayesian Inference
Preprint
Abstract
We investigate the problem of reliably assessing group fairness when labeled examples are few but unlabeled examples are plentiful. We propose a general Bayesian framework that can augment labeled data with unlabeled data to produce more accurate and lower-variance estimates compared to methods based on labeled data alone. Our approach estimates calibrated scores for unlabeled examples in each group using a hierarchical latent variable model conditioned on labeled examples. This in turn allows for inference of posterior distributions with associated notions of uncertainty for a variety of group fairness metrics. We demonstrate that our approach leads to significant and consistent reductions in estimation error across multiple well-known fairness datasets, sensitive attributes, and predictive models. The results show the benefits of using both unlabeled data and Bayesian inference in terms of assessing whether a prediction model is fair or not.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Active Bayesian Assessment of Black-Box ClassifiersProceedings of the AAAI Conference on Artificial Intelligence, 2021
- Fairness in Semi-Supervised Learning: Unlabeled Data Help to Reduce DiscriminationIEEE Transactions on Knowledge and Data Engineering, 2020
- Calibrating general posterior credible regionsBiometrika, 2018
- Fairness in Criminal Justice Risk Assessments: The State of the ArtSociological Methods & Research, 2018
- Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction InstrumentsBig Data, 2017
- A data-driven approach to predict the success of bank telemarketingDecision Support Systems, 2014
- Three naive Bayes approaches for discrimination-free classificationData Mining and Knowledge Discovery, 2010