Calibrating Classification Probabilities with Shape-Restricted Polynomial Regression
- 28 January 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 41 (8) , 1813-1827
Abstract
In many real-world classification problems, accurate prediction of membership probabilities is critical for further decision making. The probability calibration problem studies how to map scores obtained from one classification algorithm to membership probabilities. The requirement of non-decreasingness for this mapping involves an infinite number of inequality constraints, which makes its estimation computationally intractable. For the sake of this difficulty, existing methods failed to achieve four desiderata of probability calibration: universal flexibility, non-decreasingness, continuousness and computational tractability. This paper proposes a method with shape-restricted polynomial regression, which satisfies all four desiderata. In the method, the calibrating function is approximated with monotone polynomials, and the continuously-constrained requirement of monotonicity is equivalent to some semidefinite constraints. Thus, the calibration problem can be solved with tractable semidefinite programs. This estimator is both strongly and weakly universally consistent under a trivial condition. Experimental results on both artificial and real data sets clearly show that the method can greatly improve calibrating performance in terms of reliability-curve related measures.Keywords
Funding Information
- National Natural Science Foundation of China (71571163)
- Natural Science Foundation of Zhejiang Province (LY19G010001)
- Hong Kong Research Grant Council, Early Career Scheme (21202716)
- Hong Kong Innovation and Technology Commission, Innovation & Technology Fund (ITS/151/16)
- Hong Kong Research Grant Council (CityU 11302715)
This publication has 34 references indexed in Scilit:
- Calibrating predictive model estimates to support personalized medicineJournal of the American Medical Informatics Association, 2012
- Isotonic Smoothing Spline RegressionJournal of Computational and Graphical Statistics, 2008
- Nonparametric kernel regression subject to monotonicity constraintsThe Annals of Statistics, 2001
- Classification by pairwise couplingThe Annals of Statistics, 1998
- Learning Bayesian networks: The combination of knowledge and statistical dataMachine Learning, 1995
- The Monotone Smoothing of ScatterplotsTechnometrics, 1984
- Distribution-Free Maximum Likelihood Estimator of the Binary Choice ModelEconometrica, 1983
- An Empirical Distribution Function for Sampling with Incomplete InformationThe Annals of Mathematical Statistics, 1955
- Maximum Likelihood Estimates of Monotone ParametersThe Annals of Mathematical Statistics, 1955
- Point Estimates of Ordinates of Concave FunctionsJournal of the American Statistical Association, 1954