Improved Information-Theoretic Generalization Bounds for Distributed, Federated, and Iterative Learning
Open Access
Abstract
We consider information-theoretic bounds on the expected generalization error for statistical learning problems in a network setting. In this setting, there are K nodes, each with its own independent dataset, and the models from the K nodes have to be aggregated into a final centralized model. We consider both simple averaging of the models as well as more complicated multi-round algorithms. We give upper bounds on the expected generalization error for a variety of problems, such as those with Bregman divergence or Lipschitz continuous losses, that demonstrate an improved dependence of on the number of nodes. These “per node” bounds are in terms of the mutual information between the training dataset and the trained weights at each node and are therefore useful in describing the generalization properties inherent to having communication or privacy constraints at each node.
Keywords
All Related Versions
Funding Information
- National Science Foundation (CCF-1908308)
This publication has 13 references indexed in Scilit:
- Bayesian Risk With Bregman Loss: A Cramér–Rao Type Bound and Linear EstimationIEEE Transactions on Information Theory, 2021
- rTop-k: A Statistical Estimation Approach to Distributed SGDIEEE Journal on Selected Areas in Information Theory, 2020
- Information-Theoretic Bounds on the Generalization Error and Privacy Leakage in Federated Learning2020
- Tightening Mutual Information-Based Bounds on Generalization ErrorIEEE Journal on Selected Areas in Information Theory, 2020
- How Much Does Your Data Exploration Overfit? Controlling Bias via Information UsageIEEE Transactions on Information Theory, 2019
- Generalization Error Bounds for Noisy, Iterative Algorithms2018
- What Can We Learn Privately?SIAM Journal on Computing, 2011
- The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programmingUSSR Computational Mathematics and Mathematical Physics, 1967
- Randomized response: a survey technique for eliminating evasive answer bias.1965
- Randomized Response: A Survey Technique for Eliminating Evasive Answer BiasJournal of the American Statistical Association, 1965