Tuning Parameter Selection for the Adaptive Lasso Using ERIC
Authors:
- Francis K. C. HuiFHFrancis K. C. Hui
- David I. WartonDWDavid I. Warton
- Scott D. FosterSFScott D. Foster
- 2 January 2015
- journal article
- research article
- Published by Taylor & Francis in Journal of the American Statistical Association
- Vol. 110 (509) , 262-269
Abstract
The adaptive Lasso is a commonly applied penalty for variable selection in regression modeling. Like all penalties though, its performance depends critically on the choice of the tuning parameter. One method for choosing the tuning parameter is via information criteria, such as those based on AIC and BIC. However, these criteria were developed for use with unpenalized maximum likelihood estimators, and it is not clear that they take into account the effects of penalization. In this article, we propose the extended regularized information criterion (ERIC) for choosing the tuning parameter in adaptive Lasso regression. ERIC extends the BIC to account for the effect of applying the adaptive Lasso on the bias-variance tradeoff. This leads to a criterion whose penalty for model complexity is itself a function of the tuning parameter. We show the tuning parameter chosen by ERIC is selection consistent when the number of variables grows with sample size, and that this consistency holds in a wider range of contexts compared to using BIC to choose the tuning parameter. Simulation show that ERIC can significantly outperform BIC and other information criteria proposed (for choosing the tuning parameter) in selecting the true model. For ultra high-dimensional data (p > n), we consider a two-stage approach combining sure independence screening with adaptive Lasso regression using ERIC, which is selection consistent and performs strongly in simulation. Supplementary materials for this article are available online.Keywords
This publication has 29 references indexed in Scilit:
- Tuning Parameter Selection in High Dimensional Penalized LikelihoodJournal of the Royal Statistical Society Series B: Statistical Methodology, 2012
- Statistics for High-Dimensional Data2011
- Sure independence screening in generalized linear models with NP-dimensionalityThe Annals of Statistics, 2010
- Sure Independence Screening for Ultrahigh Dimensional Feature SpaceJournal of the Royal Statistical Society Series B: Statistical Methodology, 2008
- Extended Bayesian information criteria for model selection with large model spacesBiometrika, 2008
- Nonconcave penalized likelihood with a diverging number of parametersThe Annals of Statistics, 2004
- Least angle regressionThe Annals of Statistics, 2004
- Variable Selection via Nonconcave Penalized Likelihood and its Oracle PropertiesJournal of the American Statistical Association, 2001
- A Statistical View of Some Chemometrics Regression ToolsTechnometrics, 1993
- A new look at the statistical model identificationIEEE Transactions on Automatic Control, 1974