Design and Analysis of Uplink and Downlink Communications for Federated Learning
Top Cited Papers
- 2 December 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal on Selected Areas in Communications
- Vol. 39 (7) , 2150-2167
Abstract
Communication has been known to be one of the primary bottlenecks of federated learning (FL), and yet existing studies have not addressed the efficient communication design, particularly in wireless FL where both uplink and downlink communications have to be considered. In this paper, we focus on the design and analysis of physical layer quantization and transmission methods for wireless FL. We answer the question of what and how to communicate between clients and the parameter server and evaluate the impact of the various quantization and transmission options of the updated model on the learning performance. We provide new convergence analysis of the wellknown FEDAVG under non-i.i.d. dataset distributions, partial clients participation, and finite-precision quantization in uplink and downlink communications. These analyses reveal that, in order to achieve an O(1/T ) convergence rate with quantization, transmitting the weight requires increasing the quantization level at a logarithmic rate, while transmitting the weight differential can keep a constant quantization level. Comprehensive numerical evaluation on various real-world datasets reveals that the benefit of a FL-tailored uplink and downlink communication design is enormous – a carefully designed quantization and transmission achieves more than 98% of the floating-point baseline accuracy with fewer than 10% of the baseline bandwidth, for majority of the experiments on both i.i.d. and non-i.i.d. datasets. In particular, 1-bit quantization (3:1% of the floating-point baseline bandwidth) achieves 99:8% of the floating-point baseline accuracy at almost the same convergence rate on MNIST, representing the best known bandwidth-accuracy tradeoff to the best of the authors’ knowledge.Keywords
All Related Versions
Funding Information
- State’s Key Project of Research and Development Plan (2019YFE0196400)
- Guangdong Research and Development Project in Key Areas (2019B010158001)
- Guangdong Provincial Special Fund for Modern Agriculture Industry Technology Innovation Teams (2020KJ122)
- Research Grant from Kneron Inc
This publication has 16 references indexed in Scilit:
- A Survey on Distributed Machine LearningACM Computing Surveys, 2020
- Federated Learning Over Wireless Fading ChannelsIEEE Transactions on Wireless Communications, 2020
- Toward an Intelligent Edge: Wireless Communication Meets Machine LearningIEEE Communications Magazine, 2020
- A Hybrid Approach to Privacy-Preserving Federated Learning2019
- Broadband Analog Aggregation for Low-Latency Federated Edge LearningIEEE Transactions on Wireless Communications, 2019
- An Exact Quantized Decentralized Gradient Descent AlgorithmIEEE Transactions on Signal Processing, 2019
- Adaptive Federated Learning in Resource Constrained Edge Computing SystemsIEEE Journal on Selected Areas in Communications, 2019
- Fundamentals of Wireless Communication2005
- Design of fixed-point iterative decoders for concatenated codes with interleaversIEEE Journal on Selected Areas in Communications, 2001
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998