Using reference models in variable selection
Authors:
Open Access
- 14 May 2022
- journal article
- research article
- Published by Springer Nature in Computational Statistics
- Vol. 38 (1) , 349-371
Abstract
Variable selection, or more generally, model reduction is an important aspect of the statistical workflow aiming to provide insights from data. In this paper, we discuss and demonstrate the benefits of using a reference model in variable selection. A reference model acts as a noise-filter on the target variable by modeling its data generating mechanism. As a result, using the reference model predictions in the model selection procedure reduces the variability and improves stability, leading to improved model selection performance. Assuming that a Bayesian reference model describes the true distribution of future data well, the theoretically preferred usage of the reference model is to project its predictive distribution to a reduced model, leading to projection predictive variable selection approach. We analyse how much the great performance of the projection predictive variable is due to the use of reference model and show that other variable selection methods can also be greatly improved by using the reference model as target instead of the original data. In several numerical experiments, we investigate the performance of the projective prediction approach as well as alternative variable selection methods with and without reference models. Our results indicate that the use of reference models generally translates into better and more stable variable selection.Keywords
All Related Versions
Funding Information
- Research Council of Finland (298742, 313122)
- Teknologiateollisuuden 100-Vuotisjuhlasäätiö (70007503)
This publication has 28 references indexed in Scilit:
- Comparison of Bayesian predictive methods for model selectionStatistics and Computing, 2016
- ggplot22016
- Hierarchical Bayesian formulations for selecting variables in regression modelsStatistics in Medicine, 2012
- A survey of Bayesian predictive methods for model assessment, selection and comparisonStatistics Surveys, 2012
- Tweedie’s Formula and Selection BiasJournal of the American Statistical Association, 2011
- “Preconditioning” for feature selection and regression in high-dimensional problemsThe Annals of Statistics, 2008
- Prediction by Supervised Principal ComponentsJournal of the American Statistical Association, 2006
- Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequencesThe Annals of Statistics, 2004
- Variable selection in qualitative models via an entropic explanatory powerJournal of Statistical Planning and Inference, 2003
- A New Look at the Statistical Model Identification1974