Testing whether a Learning Procedure is Calibrated
Preprint
Abstract
A learning procedure takes as input a dataset and performs inference for the parameters $\theta$ of a model that is assumed to have given rise to the dataset. Here we consider learning procedures whose output is a probability distribution, representing uncertainty about $\theta$ after seeing the dataset. Bayesian inference is a prime example of such a procedure, but one can also construct other learning procedures that return distributional output. This paper studies conditions for a learning procedure to be considered calibrated, in the sense that the true data-generating parameters are plausible as samples from its distributional output. A learning procedure whose inferences and predictions are systematically over- or under-confident will fail to be calibrated. On the other hand, a learning procedure that is calibrated need not be statistically efficient. A hypothesis-testing framework is developed in order to assess, using simulation, whether a learning procedure is calibrated. Several vignettes are presented to illustrate different aspects of the framework.
Keywords
All Related Versions
This publication has 51 references indexed in Scilit:
- Probabilistic solutions to ordinary differential equations as nonlinear Bayesian filtering: a new perspectiveStatistics and Computing, 2019
- A probabilistic model for the numerical solution of initial value problemsStatistics and Computing, 2018
- Statistical analysis of differential equations: introducing probability measures on numerical solutionsStatistics and Computing, 2016
- Non-reversible Metropolis-HastingsStatistics and Computing, 2015
- Data Assimilation2015
- Markov Chains and Stochastic Stability2009
- Robust Statistics2009
- An overview of robust Bayesian analysisTEST, 1994
- Bayesian Theory1994
- Consistency and robustness of PDAF for target tracking in cluttered environmentsAutomatica, 1983