Classifier Calibration: A survey on how to assess and improve predicted class probabilities
Preprint
Abstract
This paper provides both an introduction to and a detailed overview of the principles and practice of classifier calibration. A well-calibrated classifier correctly quantifies the level of uncertainty or confidence associated with its instance-wise predictions. This is essential for critical applications, optimal decision making, cost-sensitive classification, and for some types of context change. Calibration research has a rich history which predates the birth of machine learning as an academic field by decades. However, a recent increase in the interest on calibration has led to new methods and the extension from binary to the multiclass setting. The space of options and issues to consider is large, and navigating it requires the right set of concepts and tools. We provide both introductory material and up-to-date technical details of the main concepts and methods, including proper scoring rules and other evaluation metrics, visualisation approaches, a comprehensive account of post-hoc calibration methods for binary and multiclass classification, and several advanced topics.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- On the usefulness of the fit-on-test view on evaluating calibration of classifiersMachine Learning, 2025
- Algorithmic Learning in a Random World2022
- Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methodsMachine Learning, 2021
- Fast Calibrated Additive Quantile RegressionJournal of the American Statistical Association, 2020
- Characterizing Sources of Uncertainty to Proxy Calibration and Disambiguate Annotator and Data Bias2019
- Calibrating Classification Probabilities with Shape-Restricted Polynomial RegressionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2019
- Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertaintyInformation Sciences, 2014
- Multinomial goodness‐of‐fit tests for logistic regression modelsStatistics in Medicine, 2008
- PAV and the ROC convex hullMachine Learning, 2007
- FORECASTING PRECIPITATION IN PERCENTAGES OF PROBABILITY.Monthly Weather Review, 1920