An Information-Theoretic View of Generalization via Wasserstein Distance
Authors:
- 1 July 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 577-581
Abstract
We capitalize on the Wasserstein distance to obtain two information-theoretic bounds on the generalization error of learning algorithms. First, we specialize the Wasserstein distance into total variation, by using the discrete metric. In this case we derive a generalization bound and, from a strong data-processing inequality, show how to narrow the bound by adding Gaussian noise to the output hypothesis. Second, we consider the Wasserstein distance under a generic metric. In this case we derive a generalization bound by exploiting the geometric nature of the Kantorovich-Rubinstein duality theorem. We illustrate the use of these bounds with examples. Our bounds can handle certain cases in which existing bounds via mutual information fail.Keywords
This publication has 9 references indexed in Scilit:
- Generalization error bounds using Wasserstein distances2018
- Computable Bounds on the Exploration Bias2018
- Dependence measures bounding the exploration bias for general measurements2017
- Information-theoretic analysis of stability and bias of learning algorithms2016
- Dissipation of Information in Channels With Input ConstraintsIEEE Transactions on Information Theory, 2015
- Understanding Machine Learning2014
- Concentration Inequalities2013
- On Choosing and Bounding Probability MetricsInternational Statistical Review, 2002
- Some PAC-Bayesian TheoremsMachine Learning, 1999