Far from Asymptopia: Unbiased High-Dimensional Inference Cannot Assume Unlimited Data
Open Access
Abstract
Inference from limited data requires a notion of measure on parameter space, which is most explicit in the Bayesian framework as a prior distribution. Jeffreys prior is the best-known uninformative choice, the invariant volume element from information geometry, but we demonstrate here that this leads to enormous bias in typical high-dimensional models. This is because models found in science typically have an effective dimensionality of accessible behaviors much smaller than the number of microscopic parameters. Any measure which treats all of these parameters equally is far from uniform when projected onto the sub-space of relevant parameters, due to variations in the local co-volume of irrelevant directions. We present results on a principled choice of measure which avoids this issue and leads to unbiased posteriors by focusing on relevant parameters. This optimal prior depends on the quantity of data to be gathered, and approaches Jeffreys prior in the asymptotic limit. However, for typical models, this limit cannot be justified without an impossibly large increase in the quantity of data, exponential in the number of microscopic parameters.Keywords
All Related Versions
Funding Information
- Simons Investigator Award (PHY-1607611)
- NSF (PHY-1607611)
- Simons Foundation (PHY-1607611)
This publication has 45 references indexed in Scilit:
- Correlations in ion channel expression emerge from homeostatic tuning rulesProceedings of the National Academy of Sciences, 2013
- Ensemble samplers with affine invarianceCommunications in Applied Mathematics and Computational Science, 2010
- Sloppiness, robustness, and evolvability in systems biologyCurrent Opinion in Biotechnology, 2008
- Statistical mechanical approaches to models with many poorly known parametersPhysical Review E, 2003
- The Selection of Prior Distributions by Formal RulesJournal of the American Statistical Association, 1996
- Information-theoretic asymptotics of Bayes methodsIEEE Transactions on Information Theory, 1990
- A foundation of information geometryElectronics and Communications in Japan (Part I: Communications), 1983
- Estimating the Dimension of a ModelThe Annals of Statistics, 1978
- A new look at the statistical model identificationIEEE Transactions on Automatic Control, 1974
- Computation of channel capacity and rate-distortion functionsIEEE Transactions on Information Theory, 1972