Prediction in M-complete Problems with Limited Sample Size
Authors:
- Jennifer Lynn ClarkeJCJennifer Lynn Clarke
- Bertrand ClarkeBCBertrand Clarke
- Chi-Wai YuCYChi-Wai Yu
Open Access
- 1 September 2013
- journal article
- Published by Institute of Mathematical Statistics in Bayesian Analysis
- Vol. 8 (3) , 647-690
Abstract
We define a new Bayesian predictor called the posterior weighted median (PWM) and compare its performance to several other predictors including the Bayes model average under squared error loss, the Barbieri-Berger median model predictor, the stacking predictor, and the model average predictor based on Akaike’s information criterion. We argue that PWM generally gives better performance than other predictors over a range of $\mathcal{M}$-complete problems. This range is between the $\mathcal{M}$-closed-$\mathcal{M}$-complete boundary and the $\mathcal{M}$-complete-$\mathcal{M}$-open boundary. Indeed, as a problem gets closer to $\mathcal{M}$-open, it seems that $\mathcal{M}$-complete predictive methods begin to break down. Our comparisons rest on extensive simulations and real data examples. As a separate issue, we introduce the concepts of the ‘Bail out effect’ and the ‘Bail in effect’. These occur when a predictor gives not just poor results but defaults to the simplest model (‘bails out’) or to the most complex model (‘bails in’) on the model list. Either can occur in $\mathcal{M}$-complete problems when the complexity of the data generator is too high for the predictor scheme to accommodate.Keywords
This publication has 22 references indexed in Scilit:
- Prequential analysis of complex data with adaptive model reselectionStatistical Analysis and Data Mining: An ASA Data Science Journal, 2009
- On boosting kernel regressionJournal of Statistical Planning and Inference, 2008
- Improvement over bayes prediction in small samples in the presence of model uncertaintyCanadian Journal of Statistics, 2004
- Optimal predictive model selectionThe Annals of Statistics, 2004
- Bayesian model averaging: a tutorial (with comments by M. Clyde, David Draper and E. I. George, and a rejoinder by the authorsStatistical Science, 1999
- On Efficient Point Prediction SystemsJournal of the Royal Statistical Society Series B: Statistical Methodology, 1998
- Minimum complexity density estimationIEEE Transactions on Information Theory, 1991
- Asymptotic mean efficiency of a selection of regression variablesAnnals of the Institute of Statistical Mathematics, 1983
- An optimal selection of regression variablesBiometrika, 1981
- Inconsistency of the AIC rule for estimating the order of autoregressive modelsIEEE Transactions on Automatic Control, 1980