Calibrating Bayesian inference
Open Access
- 30 July 2026
- journal article
- research article
- Published by Wiley in British Journal of Mathematical and Statistical Psychology
Abstract
Bayesian statistics has gained popularity in psychological research due to its intuitive uncertainty quantification and convenient information‐updating rules. In many applications, however, prior distributions are introduced merely as instruments to facilitate computation, rather than as representations of genuine subjective belief. Consequently, relying on standard Bayesian justifications for inferential procedures becomes conceptually ungrounded. In this paper, we recommend evaluating finite‐sample performance over repeated sampling of data and parameters as an alternative justification for “pragmatic Bayes.” We demonstrate a key vulnerability in the usual posterior‐based inference: when analysts' chosen prior distribution mismatches the true parameter‐generating process, Bayesian inference can be misleading in the long run. Given that this true process is rarely known in practice, we propose a safer alternative: calibrating Bayesian credible regions to achieve frequentist validity. This latter criterion is stronger and guarantees validity of Bayesian inference regardless of the underlying parameter‐generating mechanism. To solve the calibration problem in practice, we propose a novel stochastic approximation algorithm. Monte Carlo experiments are conducted and reported, in which we observe that uncalibrated Bayesian inference can be liberal under certain parameter‐generating scenarios, whereas our calibrated solution consistently maintain validity. We also illustrate the proposed calibration procedure using a real‐data example involving location‐scale regression.Keywords
All Related Versions
Funding Information
- National Science Foundation (DMS‐2113404)
- National Science Foundation (2210337)
- National Science Foundation (2515303)
This publication has 65 references indexed in Scilit:
- A Riemannian Optimization Algorithm for Joint Maximum Likelihood Estimation of High-Dimensional Exploratory Item Factor AnalysisPsychometrika, 2020
- Safe probabilityJournal of Statistical Planning and Inference, 2018
- Frequency-calibrated belief functions: Review and new insightsInternational Journal of Approximate Reasoning, 2018
- Vector Generalized Linear and Additive Models2015
- Possibility theory and statistical reasoningComputational Statistics & Data Analysis, 2006
- Probability Matching Priors: Higher Order Asymptotics2004
- Structural Equations with Latent Variables1989
- Statistical Decision Theory and Bayesian Analysis1985
- Fuzzy sets as a basis for a theory of possibilityFuzzy Sets and Systems, 1978
- A General Approach to Confirmatory Maximum Likelihood Factor AnalysisPsychometrika, 1969