Mitigating Algorithmic Bias Through Probability Calibration: A Case Study on Lead Generation Data
Open Access
- 3 July 2025
- journal article
- research article
- Published by MDPI AG in Mathematics
- Vol. 13 (13) , 2183
Abstract
Probability calibration is commonly utilized to enhance the reliability and interpretability of probabilistic classifiers, yet its potential for reducing algorithmic bias remains under-explored. In this study, the role of probability calibration techniques in mitigating bias associated with sensitive attributes, specifically country of origin, within binary classification models is investigated. Using a real-world lead-generation 2853 × 8 matrix dataset characterized by substantial class imbalance, with the positive class representing 1.4% of observations, several binary classification models were evaluated and the best-performing model was selected as the baseline for further analysis. The evaluated models included Binary Logistic Regression with polynomial degrees of 1, 2, 3, and 4, Random Forest, and XGBoost classification algorithms. Three widely used calibration methods, Platt scaling, isotonic regression, and temperature scaling, were then used to assess their impact on both probabilistic accuracy and fairness metrics of the best-performing model. The findings suggest that post hoc calibration can effectively reduce the influence of sensitive features on predictions by improving fairness without compromising overall classification performance. This study demonstrates the practical value of incorporating calibration as a straightforward and effective fairness intervention within machine learning workflows.Keywords
Funding Information
- Ministry of Science, Technological Development and Innovation (451-03-136/2025-03/200156)
This publication has 39 references indexed in Scilit:
- The effect of resampling techniques on the performances of machine learning clinical risk prediction models in the setting of severe class imbalance: development and internal validation in a retrospective cohortDiscover Artificial Intelligence, 2024
- Calibrating machine learning approaches for probability estimation: A comprehensive comparisonStatistics in Medicine, 2023
- Fairness of artificial intelligence in healthcare: review and recommendationsJapanese Journal of Radiology, 2023
- The harm of class imbalance corrections for risk prediction models: illustration and simulation using logistic regressionJournal of the American Medical Informatics Association, 2022
- Algorithmic bias in machine learning-based marketing modelsJournal of Business Research, 2022
- Empirical observation of negligible fairness–accuracy trade-offs in machine learning for public policyNature Machine Intelligence, 2021
- A comparative analysis of gradient boosting algorithmsArtificial Intelligence Review, 2020
- Interpretation of machine learning models using shapley values: application to compound potency and multi-target activity predictionsJournal of Computer-Aided Molecular Design, 2020
- Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction InstrumentsBig Data, 2017
- Analyzing the presence of noise in multi-class problems: alleviating its influence with the One-vs-One decompositionKnowledge and Information Systems, 2012