A small-sample Bayesian information criterion that does not overstate the evidence, with an application to calibrating p-values from likelihood-ratio tests
Open Access
- 22 March 2025
- journal article
- research article
- Published by Springer Nature in Statistical Papers
- Vol. 66 (3) , 1-17
Abstract
This paper proposes a simple correction to the Bayesian information criterion (BIC) for small samples to ensure that it neither overstates nor understates the evidence against a null hypothesis or other tested model. The new correction raises the likelihood ratio in the BIC to the power of 1 minus the reciprocal of the sample size ( $$1-1/\textrm{n}, \textrm{n}>1$$ ). That is equivalent to multiplying the loglikelihood term of the BIC by a factor of $$1-1/\textrm{n}.$$ The correction is applied to the problem of calibrating p-values by transforming them to estimated Bayes factors. The corresponding calibration in the most common case is simply sqrt(n)/exp((1−1/n)*qchisq(1−p,df=1)/2) in R syntax, where the p-value is from a likelihood-ratio test. That intersects the class of betting scores called e-values and, more specifically, admissible calibrators. While all admissible calibrators neither overstate nor understate the evidence against the null hypothesis, previous admissible calibrators are not model-selection consistent since they do not increasingly favor the null hypothesis when it is true. The proposed calibrator is consistent under general conditions, for its corrected BIC is asymptotically equivalent to the BIC.
Keywords
Funding Information
- University of North Carolina at Greensboro
- University of North Carolina at Greensboro
This publication has 50 references indexed in Scilit:
- Statistical significance and its critics: practicing damaging science, or damaging scientific practice?Synthese, 2022
- A measure of evidence based on the likelihood-ratio statisticsStatistical Papers, 2022
- Game‐Theoretic Foundations for Probability and Finance2019
- Log‐symmetric regression models: information criteria and application to movie business and industry data with economic implicationsApplied Stochastic Models in Business and Industry, 2019
- The evidential statistical paradigm in geneticsGenetic Epidemiology, 2018
- The Jeffreys–Lindley paradox and discovery criteria in high energy physicsSynthese, 2014
- The Bayesian information criterion: background, derivation, and applicationsWIREs Computational Statistics, 2011
- The ongoing tyranny of statistical significance testing in biomedical researchEuropean Journal of Epidemiology, 2010
- Likelihood methods for measuring statistical evidenceStatistics in Medicine, 2002
- Probability and Finance2001