A Safe Hosmer-Lemeshow Test
Open Access
- 18 December 2023
- journal article
- research article
- Published by New England Statistical Society in The New England Journal of Statistics in Data Science
- p. 175-189
Abstract
This article proposes an alternative to the Hosmer-Lemeshow (HL) test for evaluating the calibration of probability forecasts for binary events. The approach is based on e-values, a new tool for hypothesis testing. An e-value is a random variable with expected value less or equal to one under a null hypothesis. Large e-values give evidence against the null hypothesis, and the multiplicative inverse of an e-value is a p-value. Our test uses online isotonic regression to estimate the calibration curve as a ‘betting strategy’ against the null hypothesis. We show that the test has power against essentially all alternatives, which makes it theoretically superior to the HL test and at the same time resolves the well-known instability problem of the latter. A simulation study shows that a feasible version of the proposed eHL test can detect slight miscalibrations in practically relevant sample sizes, but trades its universal validity and power guarantees against a reduced empirical power compared to the HL test in a classical simulation setup. We illustrate our test on recalibrated predictions for credit card defaults during the Taiwan credit card crisis, where the classical HL test delivers equivocal results.Keywords
All Related Versions
This publication has 34 references indexed in Scilit:
- A comparison of the Hosmer–Lemeshow, Pigeon–Heyse, and Tsiatis goodness-of-fit tests for binary logistic regression under two grouping methodsCommunications in Statistics - Simulation and Computation, 2015
- Safe Probability: Restricted Conditioning and Extended MarginalizationLecture Notes in Computer Science, 2013
- Necrotizing Cutaneous Mucormycosis after a Tornado in Joplin, Missouri, in 2011New England Journal of Medicine, 2012
- The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clientsExpert Systems with Applications, 2009
- Increasing the power: A practical approach to goodness-of-fit test for logistic regression models with continuous predictorsComputational Statistics & Data Analysis, 2007
- Global goodness‐of‐fit tests in logistic regression with sparse dataStatistics in Medicine, 2002
- Goodness‐of‐fit processes for logistic regression: simulation resultsStatistics in Medicine, 2002
- Goodness of fit tests for the multiple logistic regression modelCommunications in Statistics - Theory and Methods, 1980
- Conditional Expectation Given A $\sigma$-Lattice and ApplicationsThe Annals of Mathematical Statistics, 1965
- An Empirical Distribution Function for Sampling with Incomplete InformationThe Annals of Mathematical Statistics, 1955