Field-aware Calibration: A Simple and Empirically Strong Method for Reliable Probabilistic Predictions
Authors:
Open Access
- 20 April 2020
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 729-739
Abstract
It is often observed that the probabilistic predictions given by a machine learning model can disagree with averaged actual outcomes on specific subsets of data, which is also known as the issue of miscalibration. It is responsible for the unreliability of practical machine learning systems. For example, in online advertising, an ad can receive a click-through rate prediction of 0.1 over some population of users where its actual click rate is 0.15. In such cases, the probabilistic predictions have to be fixed before the system can be deployed. In this paper, we first introduce a new evaluation metric named field-level calibration error that measures the bias in predictions over the sensitive input field that the decision-maker concerns. We show that existing post-hoc calibration methods have limited improvements in the new field-level metric and other non-calibration metrics such as the AUC score. To this end, we propose Neural Calibration, a simple yet powerful post-hoc calibration method that learns to calibrate by making full use of the field-aware information over the validation set. We present extensive experiments on five large-scale datasets. The results showed that Neural Calibration significantly improves against uncalibrated predictions in common metrics such as the negative log-likelihood, Brier score and AUC, as well as the proposed field-level calibration error.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Warm Up Cold-start Advertisements2019
- Calibration2018
- Assessing calibration of prognostic risk scoresStatistical Methods in Medical Research, 2016
- Practical Lessons from Predicting Clicks on Ads at Facebook2014
- Ad click prediction2013
- Fairness-aware Learning through Regularization Approach2011
- Calibration of Machine Learning Models2010
- Predicting good probabilities with supervised learning2005
- Bayesian Isotonic Regression and Trend AnalysisBiometrics, 2004
- Transforming classifier scores into accurate multiclass probability estimates2002