Information-Theoretic Bounds on the Generalization Error and Privacy Leakage in Federated Learning
- 1 May 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-5
Abstract
Machine learning algorithms operating on mobile networks can be characterized into three different categories. First is the classical situation in which the end-user devices send their data to a central server where this data is used to train a model. Second is the distributed setting in which each device trains its own model and send its model parameters to a central server where these model parameters are aggregated to create one final model. Third is the federated learning setting in which, at any given time t, a certain number of active end users train with their own local data along with feedback provided by the central server and then send their newly estimated model parameters to the central server. The server, then, aggregates these new parameters, updates its own model, and feeds the updated parameters back to all the end users, continuing this process until it converges.The main objective of this work is to provide an information-theoretic framework for all of the aforementioned learning paradigms. Moreover, using the provided framework, we develop upper and lower bounds on the generalization error together with bounds on the privacy leakage in the classical, distributed and federated learning settings.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Tightening Mutual Information Based Bounds on Generalization Error2019
- An Information-Theoretic View of Generalization via Wasserstein Distance2019
- Generalization Error Bounds for Noisy, Iterative Algorithms2018
- Dependence measures bounding the exploration bias for general measurements2017
- Information-theoretic analysis of stability and bias of learning algorithms2016
- Differential Privacy as a Mutual Information Constraint2016
- Bayesian Differential Privacy on Correlated Data2015
- Distributed stochastic optimization and learning2014
- Calibrating Noise to Sensitivity in Private Data Analysis2006