Learning models with uniform performance via distributionally robust optimization
Authors:
Top Cited Papers
- 1 June 2021
- journal article
- research article
- Published by Institute of Mathematical Statistics in The Annals of Statistics
- Vol. 49 (3) , 1378-1406
Abstract
A common goal in statistics and machine learning is to learn models that can perform well against distributional shifts, such as latent heterogeneous subpopulations, unknown covariate shifts or unmodeled temporal effects. We develop and analyze a distributionally robust stochastic optimization (DRO) framework that learns a model providing good performance against perturbations to the data-generating distribution. We give a convex formulation for the problem, providing several convergence guarantees. We prove finite-sample minimax upper and lower bounds, showing that distributional robustness sometimes comes at a cost in convergence rates. We give limit theorems for the learned parameters, where we fully specify the limiting distribution so that confidence intervals can be computed. On real tasks including generalizing to unknown subpopulations, fine-grained recognition and providing good tail performance, the distributionally robust approach often exhibits improved performance.Keywords
All Related Versions
This publication has 82 references indexed in Scilit:
- A framework for optimization under ambiguityAnnals of Operations Research, 2010
- Direct Density Ratio Estimation for Large-scale Covariate Shift AdaptationJournal of Information Processing, 2009
- Higher moment coherent risk measuresQuantitative Finance, 2007
- Ambiguity in portfolio selectionQuantitative Finance, 2007
- Regularization and Variable Selection Via the Elastic NetJournal of the Royal Statistical Society Series B: Statistical Methodology, 2005
- Minimax optimal control of stochastic uncertain systems with relative entropy constraintsIEEE Transactions on Automatic Control, 2000
- Coherent Measures of RiskMathematical Finance, 1999
- A new look at independenceThe Annals of Probability, 1996
- Epi‐consistency of convex stochastic programsStochastics and Stochastic Reports, 1991
- The Integral of a Symmetric Unimodal Function over a Symmetric Convex Set and Some Probability InequalitiesProceedings of the American Mathematical Society, 1955