Topology-aware Robust Optimization for Out-of-distribution Generalization
Preprint
Abstract
Out-of-distribution (OOD) generalization is a challenging machine learning problem yet highly desirable in many high-stake applications. Existing methods suffer from overly pessimistic modeling with low generalization confidence. As generalizing to arbitrary test distributions is impossible, we hypothesize that further structure on the topology of distributions is crucial in developing strong OOD resilience. To this end, we propose topology-aware robust optimization (TRO) that seamlessly integrates distributional topology in a principled optimization framework. More specifically, TRO solves two optimization objectives: (1) Topology Learning which explores data manifold to uncover the distributional topology; (2) Learning on Topology which exploits the topology to constrain robust optimization for tightly-bounded generalization risks. We theoretically demonstrate the effectiveness of our approach and empirically show that it significantly outperforms the state of the arts in a wide range of tasks including classification, regression, and semantic segmentation. Moreover, we empirically find the data-driven distributional topology is consistent with domain knowledge, enhancing the explainability of our approach.Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- Out-of-Domain Generalization From a Single Source: An Uncertainty Quantification ApproachIEEE Transactions on Pattern Analysis and Machine Intelligence, 2022
- Deeper, Broader and Artier Domain Generalization2017
- Hölder–Lipschitz Norms and Their Duals on Spaces with Semigroups, with Applications to Earth Mover’s DistanceJournal of Fourier Analysis and Applications, 2015
- Distributionally Robust Optimization Under Moment Uncertainty with Application to Data-Driven ProblemsOperations Research, 2010
- Robust Stochastic Approximation Approach to Stochastic ProgrammingSIAM Journal on Optimization, 2009