FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning
Authors:
Top Cited Papers
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 16323-16332
Abstract
Federated learning (FL) has recently attracted increasing attention from academia and industry, with the ultimate goal of achieving collaborative training under privacy and communication constraints. Existing iterative model averaging based FL algorithms require a large number of communication rounds to obtain a well-performed model due to extremely unbalanced and non-i.i.d data partitioning among different clients. Thus, we propose FedDM to build the global training objective from multiple local surrogate functions, which enables the server to gain a more global view of the loss landscape. In detail, we construct synthetic sets of data on each client to locally match the loss landscape from original data through distribution matching. FedDM reduces communication rounds and improves model quality by transmitting more informative and smaller synthesized data compared with unwieldy model weights. We conduct extensive experiments on three image classification datasets, and show that our method outperforms other FL counterparts in terms of efficiency and model performance given a limited number of communication rounds. Moreover, we demonstrate that FedDM can be adapted to preserve differential privacy with Gaussian mechanism and train a better model under the same privacy budget.Keywords
All Related Versions
Funding Information
- NSF (IIS-2008173,IIS-2048280)
This publication has 15 references indexed in Scilit:
- Communication-Efficient Federated Deep Learning With Layerwise Asynchronous Model Update and Temporally Weighted AggregationIEEE Transactions on Neural Networks and Learning Systems, 2019
- Robust and Communication-Efficient Federated Learning From Non-i.i.d. DataIEEE Transactions on Neural Networks and Learning Systems, 2019
- The Composition Theorem for Differential PrivacyIEEE Transactions on Information Theory, 2017
- Deep Learning with Differential Privacy2016
- Deep Residual Learning for Image Recognition2016
- The Algorithmic Foundations of Differential PrivacyFoundations and Trends® in Theoretical Computer Science, 2014
- A firm foundation for private data analysisCommunications of the ACM, 2011
- Privacy integrated queries2009
- Differential privacy and robust statistics2009
- Our Data, Ourselves: Privacy Via Distributed Noise Generation2006