Cross-Validation With Confidence
Author:
- 1 October 2019
- journal article
- research article
- Published by Taylor & Francis in Journal of the American Statistical Association
- Vol. 115 (532) , 1978-1997
Abstract
Cross-validation is one of the most popular model and tuning parameter selection methods in statistics and machine learning. Despite its wide applicability, traditional cross-validation methods tend to overfit, due to the ignorance of the uncertainty in the testing sample. We develop a novel statistically principled inference tool based on cross-validation that takes into account the uncertainty in the testing sample. This method outputs a set of highly competitive candidate models containing the optimal one with guaranteed probability. As a consequence, our method can achieve consistent variable selection in a classical linear regression setting, for which existing cross-validation methods require unconventional split ratios. When used for tuning parameter selection, the method can provide an alternative trade-off between prediction accuracy and model interpretability than existing variants of cross-validation. We demonstrate the performance of the proposed method in several simulated and real data examples.Keywords
Funding Information
- NSF (DMS-1407771, DMS-1553884)
This publication has 36 references indexed in Scilit:
- The Bernstein–Orlicz norm and deviation inequalitiesProbability Theory and Related Fields, 2012
- Stability SelectionJournal of the Royal Statistical Society Series B: Statistical Methodology, 2010
- Simultaneous analysis of Lasso and Dantzig selectorThe Annals of Statistics, 2009
- Lasso-type recovery of sparse representations for high-dimensional dataThe Annals of Statistics, 2009
- Fence methods for mixed model selectionThe Annals of Statistics, 2008
- High-dimensional generalized linear models and the lassoThe Annals of Statistics, 2008
- Linear Model Selection by Cross-validationJournal of the American Statistical Association, 1993
- Asymptotic Optimality for $C_p, C_L$, Cross-Validation and Generalized Cross-Validation: Discrete Index SetThe Annals of Statistics, 1987
- The Predictive Sample Reuse Method with ApplicationsJournal of the American Statistical Association, 1975
- The Relationship Between Variable Selection and Data Agumentation and a Method for PredictionTechnometrics, 1974