A Comprehensive Empirical Study of Bias Mitigation Methods for Machine Learning Classifiers
Top Cited Papers
Open Access
- 27 May 2023
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 32 (4) , 1-30
Abstract
Software bias is an increasingly important operational concern for software engineers. We present a large-scale, comprehensive empirical study of 17 representative bias mitigation methods for Machine Learning (ML) classifiers, evaluated with 11 ML performance metrics (e.g., accuracy), 4 fairness metrics, and 20 types of fairness-performance tradeoff assessment, applied to 8 widely-adopted software decision tasks. The empirical coverage is much more comprehensive, covering the largest numbers of bias mitigation methods, evaluation metrics, and fairness-performance tradeoff measures compared to previous work on this important software property. We find that (1) the bias mitigation methods significantly decrease ML performance in 53% of the studied scenarios (ranging between 42%∼66% according to different ML performance metrics); (2) the bias mitigation methods significantly improve fairness measured by the 4 used metrics in 46% of all the scenarios (ranging between 24%∼59% according to different fairness metrics); (3) the bias mitigation methods even lead to decrease in both fairness and ML performance in 25% of the scenarios; (4) the effectiveness of the bias mitigation methods depends on tasks, models, the choice of protected attributes, and the set of metrics used to assess fairness and ML performance; (5) there is no bias mitigation method that can achieve the best tradeoff in all the scenarios. The best method that we find outperforms other methods in 30% of the scenarios. Researchers and practitioners need to choose the bias mitigation method best suited to their intended application scenario(s).Keywords
All Related Versions
Funding Information
- EPIC: Evolutionary Program Improvement Collaborators (741278)
- UKRI Trustworthy Autonomous Systems Node in Verifiability (EP/V026801/2)
This publication has 54 references indexed in Scilit:
- Semantics derived automatically from language corpora contain human-like biasesScience, 2017
- Certifying and Removing Disparate Impact2015
- Preliminary comparison of techniques for dealing with imbalance in software defect prediction2014
- Fairness-Aware Classifier with Prejudice Remover Regularizer2012
- Data preprocessing techniques for classification without discriminationKnowledge and Information Systems, 2011
- Three naive Bayes approaches for discrimination-free classificationData Mining and Knowledge Discovery, 2010
- Online advertising: Pay-per-view versus pay-per-clickJournal of Revenue and Pricing Management, 2004
- Machine learning in automated text categorizationACM Computing Surveys, 2002
- On the application of measurement theory in software engineeringEmpirical Software Engineering, 1996
- On a Test of Whether one of Two Random Variables is Stochastically Larger than the OtherThe Annals of Mathematical Statistics, 1947