Distributionally Robust Groupwise Regularization Estimator
Preprint
Abstract
Regularized estimators in the context of group variables have been applied successfully in model and feature selection in order to preserve interpretability. We formulate a Distributionally Robust Optimization (DRO) problem which recovers popular estimators, such as Group Square Root Lasso (GSRL). Our DRO formulation allows us to interpret GSRL as a game, in which we learn a regression parameter while an adversary chooses a perturbation of the data. We wish to pick the parameter to minimize the expected loss under any plausible model chosen by the adversary - who, on the other hand, wishes to increase the expected loss. The regularization parameter turns out to be precisely determined by the amount of perturbation on the training data allowed by the adversary. In this paper, we introduce a data-driven (statistical) criterion for the optimal choice of regularization, which we evaluate asymptotically, in closed form, as the size of the training set increases. Our easy-to-evaluate regularization formula is compared against cross-validation, showing good (sometimes superior) performance.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Robust Wasserstein profile inference and applications to machine learningJournal of Applied Probability, 2019
- The Group Square-Root Lasso: Theoretical Properties and Fast AlgorithmsIEEE Transactions on Information Theory, 2013
- Scaled sparse linear regressionBiometrika, 2012
- Square-root lasso: pivotal recovery of sparse signals via conic programmingBiometrika, 2011
- The Group Lasso for Logistic RegressionJournal of the Royal Statistical Society Series B: Statistical Methodology, 2008
- Model Selection and Estimation in Regression with Grouped VariablesJournal of the Royal Statistical Society Series B: Statistical Methodology, 2005
- Regression Shrinkage and Selection Via the LassoJournal of the Royal Statistical Society Series B: Statistical Methodology, 1996