Learning Against Distributional Uncertainty: On the Trade-off Between Robustness and Specificity
Preprint
Abstract
Trustworthy machine learning aims at combating distributional uncertainties in training data distributions compared to population distributions. Typical treatment frameworks include the Bayesian approach, (min-max) distributionally robust optimization (DRO), and regularization. However, two issues have to be raised: 1) All these methods are biased estimators of the true optimal cost; 2) the prior distribution in the Bayesian method, the radius of the distributional ball in the DRO method, and the regularizer in the regularization method are difficult to specify. This paper studies a new framework that unifies the three approaches and that addresses the two challenges mentioned above. The asymptotic properties (e.g., consistency and asymptotic normalities), non-asymptotic properties (e.g., unbiasedness and generalization error bound), and a Monte--Carlo-based solution method of the proposed model are studied. The new model reveals the trade-off between the robustness to the unseen data and the specificity to the training data.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Wasserstein Distributionally Robust Optimization and Variation RegularizationOperations Research, 2024
- Finite-Sample Guarantees for Wasserstein Distributionally Robust Optimization: Breaking the Curse of DimensionalityOperations Research, 2023
- Distributionally Robust Stochastic Optimization with Wasserstein DistanceMathematics of Operations Research, 2023
- On linear optimization over Wasserstein ballsMathematical Programming, 2021
- When can we improve on sample average approximation for stochastic optimization?Operations Research Letters, 2020
- Robust Wasserstein profile inference and applications to machine learningJournal of Applied Probability, 2019
- Cross-validation failure: Small sample sizes lead to large error barsNeuroImage, 2018
- A Bayesian Risk Approach to Data-driven Stochastic Optimization: Formulations and AsymptoticsSIAM Journal on Optimization, 2018
- On the rate of convergence in Wasserstein distance of the empirical measureProbability Theory and Related Fields, 2014
- A Bayesian Analysis of Some Nonparametric ProblemsThe Annals of Statistics, 1973