Distributional Robustness Bounds Generalization Errors
Preprint
Abstract
Bayesian methods, distributionally robust optimization methods, and regularization methods are three pillars of trustworthy machine learning combating distributional uncertainty, e.g., the uncertainty of an empirical distribution compared to the true underlying distribution. This paper investigates the connections among the three frameworks and, in particular, explores why these frameworks tend to have smaller generalization errors. Specifically, first, we suggest a quantitative definition for "distributional robustness", propose the concept of "robustness measure", and formalize several philosophical concepts in distributionally robust optimization. Second, we show that Bayesian methods are distributionally robust in the probably approximately correct (PAC) sense; in addition, by constructing a Dirichlet-process-like prior in Bayesian nonparametrics, it can be proven that any regularized empirical risk minimization method is equivalent to a Bayesian method. Third, we show that generalization errors of machine learning models can be characterized using the distributional uncertainty of the nominal distribution and the robustness measures of these machine learning models, which is a new perspective to bound generalization errors, and therefore, explain the reason why distributionally robust machine learning models, Bayesian models, and regularization models tend to have smaller generalization errors in a unified manner.Keywords
All Related Versions
This publication has 36 references indexed in Scilit:
- When can we improve on sample average approximation for stochastic optimization?Operations Research Letters, 2020
- A Comprehensive Survey of Loss Functions in Machine LearningAnnals of Data Science, 2020
- Robust Wasserstein profile inference and applications to machine learningJournal of Applied Probability, 2019
- Characterization of the equivalence of robustification and regularization in linear and matrix regressionEuropean Journal of Operational Research, 2018
- Robust empirical optimization is almost the same as mean–variance optimizationOperations Research Letters, 2018
- Distributionally Robust Control of Constrained Stochastic SystemsIEEE Transactions on Automatic Control, 2015
- On the rate of convergence in Wasserstein distance of the empirical measureProbability Theory and Related Fields, 2014
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Linear estimation in Krein spaces. I. TheoryIEEE Transactions on Automatic Control, 1996
- Stability analysis for stochastic programsAnnals of Operations Research, 1991