Testing whether a Learning Procedure is Calibrated
Preprint
Abstract
A learning procedure takes as input a dataset and performs inference for the parameters $\theta$ of a model that is assumed to have given rise to the dataset. Here we consider learning procedures whose output is a probability distribution, representing uncertainty about $\theta$ after seeing the dataset. Bayesian inference is a prime example of such a procedure but one can also construct other learning procedures that return distributional output. This paper studies conditions for a learning procedure to be considered calibrated, in the sense that the true data-generating parameters are plausible as samples from its distributional output. A learning procedure that is calibrated need not be statistically efficient and vice versa. A hypothesis-testing framework is developed in order to assess, using simulation, whether a learning procedure is calibrated. Finally, we exploit our framework to test the calibration of some learning procedures that are motivated as being approximations to Bayesian inference but are nevertheless widely used.
Keywords
All Related Versions
This publication has 37 references indexed in Scilit:
- Variational Inference: A Review for StatisticiansJournal of the American Statistical Association, 2017
- Non-reversible Metropolis-HastingsStatistics and Computing, 2015
- Data Assimilation2015
- Markov Chains and Stochastic Stability2009
- Robust Statistics2009
- Numerical maximum likelihood estimation for the g-and-k and generalized g-and-h distributionsStatistics and Computing, 2002
- An overview of robust Bayesian analysisTEST, 1994
- Bayesian Theory1994
- Consistency and robustness of PDAF for target tracking in cluttered environmentsAutomatica, 1983
- The Well-Calibrated BayesianJournal of the American Statistical Association, 1982