Tighter Expected Generalization Error Bounds via Convexity of Information Measures
Authors:
- 26 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2481-2486
Abstract
Generalization error bounds are essential to understanding machine learning algorithms. This paper presents novel expected generalization error upper bounds based on the average joint distribution between the output hypothesis and each input training sample. Multiple generalization error upper bounds based on different information measures are provided, including Wasserstein distance, total variation distance, KL divergence, and Jensen-Shannon divergence. Due to the convexity of the information measures, the proposed bounds in terms of Wasserstein distance and total variation distance are shown to be tighter than their counterparts based on individual samples in the literature. An example is provided to demonstrate the tightness of the proposed generalization error bounds.Keywords
All Related Versions
Funding Information
- Royal Society
This publication has 17 references indexed in Scilit:
- Tightening Mutual Information-Based Bounds on Generalization ErrorIEEE Journal on Selected Areas in Information Theory, 2020
- How Much Does Your Data Exploration Overfit? Controlling Bias via Information UsageIEEE Transactions on Information Theory, 2019
- An Information-Theoretic View of Generalization via Wasserstein Distance2019
- Generalization error bounds using Wasserstein distances2018
- Robustness and generalizationMachine Learning, 2011
- Optimal Transport2009
- Lautum InformationIEEE Transactions on Information Theory, 2008
- PAC-Bayesian Stochastic Model SelectionMachine Learning, 2003
- An overview of statistical learning theoryIEEE Transactions on Neural Networks, 1999
- Divergence measures based on the Shannon entropyIEEE Transactions on Information Theory, 1991