Learning Against Distributional Uncertainty: On the Trade-Off Between Robustness and Specificity
- 5 May 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Signal Processing
- Vol. 19 (7) , 1420-1435
Abstract
Trustworthy machine learning aims at combating distributional uncertainties in training data distributions compared to population distributions. Typical treatment frameworks include the Bayesian approach, (min-max) distributionally robust optimization (DRO), and regularization. However, three issues have to be raised: 1) the prior distribution in the Bayesian method and the regularizer in the regularization method are difficult to specify; 2) the DRO method tends to be overly conservative; 3) all the three methods are biased estimators of the true optimal cost. This paper studies a new framework that unifies the three approaches and addresses the three challenges above. The asymptotic properties (e.g., consistencies and asymptotic normalities), non-asymptotic properties (e.g., generalization bounds and unbiasedness), and solution methods of the proposed model are studied. The new model reveals the trade-off between the robustness to the unseen data and the specificity to the training data. Experiments on various real-world tasks validate the superiority of the proposed learning framework.Keywords
All Related Versions
Funding Information
- National Research Foundation Singapore and DSO National Laboratories through the AI Singapore Programme (AISG2RP-2020-018)
This publication has 33 references indexed in Scilit:
- On linear optimization over Wasserstein ballsMathematical Programming, 2021
- When can we improve on sample average approximation for stochastic optimization?Operations Research Letters, 2020
- A Comprehensive Survey of Loss Functions in Machine LearningAnnals of Data Science, 2020
- Cross-validation failure: Small sample sizes lead to large error barsNeuroImage, 2018
- Data-driven risk-averse stochastic optimization with Wasserstein metricOperations Research Letters, 2018
- Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulationsMathematical Programming, 2017
- On the rate of convergence in Wasserstein distance of the empirical measureProbability Theory and Related Fields, 2014
- Bootstrap Methods2011
- The Elements of Statistical Learning2009
- Weak Convergence and Empirical Processes1996