Slashing Communication Traffic in Federated Learning by Transmitting Clustered Model Updates
- 7 June 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal on Selected Areas in Communications
- Vol. 39 (8) , 2572-2589
Abstract
Federated Learning (FL) is an emerging decentralized learning framework through which multiple clients can collaboratively train a learning model. However, a major obstacle that impedes the wide deployment of FL lies in massive communication traffic. To train high dimensional machine learning models (such as CNN models), heavy communication traffic can be incurred by exchanging model updates via the Internet between clients and the parameter server (PS), implying that the network resource can be easily exhausted. Compressing model updates is an effective way to reduce the traffic amount. However, a flexible unbiased compression algorithm applicable for both uplink and downlink compression in FL is still absent from existing works. In this work, we devise the Model Update Compression by Soft Clustering (MUCSC) algorithm to compress model updates transmitted between clients and the PS. In MUCSC, it is only necessary to transmit cluster centroids and the cluster ID of each model update. Moreover, we prove that: 1) The compressed model updates are unbiased estimation of their original values so that the convergence rate by transmitting compressed model updates is unchanged; 2) MUCSC can guarantee that the influence of the compression error on the model accuracy is minimized. Then, we further propose the boosted MUCSC (B-MUCSC) algorithm, a biased compression algorithm that can achieve an extremely high compression rate by grouping insignificant model updates into a super cluster. B-MUCSC is suitable for scenarios with very scarce network resource. Ultimately, we conduct extensive experiments with the CIFAR-10 and FEMNIST datasets to demonstrate that our algorithms can not only substantially reduce the volume of communication traffic in FL, but also improve the training efficiency in practical networks.Keywords
All Related Versions
Funding Information
- National Key Research and Development Plan of China (2018YFB1800302, 2018YFB1800805)
- National Natural Science Foundation of China (61772345)
- Shenzhen Science and Technology Program (RCYX20200714114645048, JCYJ20190808142207420, GJHZ20190822095416463)
- Project of ”FANet: PCL Future Greater-Bay Area Network Facilities for Large-scale Experiments and Applications (LZC0019)
- Australia Research Council (DE180100950)
- Pearl River Young Scholars Funding of Shenzhen University
This publication has 21 references indexed in Scilit:
- Adaptive Federated Learning in Resource Constrained Edge Computing SystemsIEEE Journal on Selected Areas in Communications, 2019
- Federated Machine LearningACM Transactions on Intelligent Systems and Technology, 2019
- Randomized Distributed Mean Estimation: Accuracy vs. CommunicationFrontiers in Applied Mathematics and Statistics, 2018
- Two-Stream Federated Learning: Reduce the Communication Costs2018
- Federated Learning Based Proactive Content Caching in Edge Computing2018
- Federated Learning for Ultra-Reliable Low-Latency V2V Communications2018
- Federated learning of predictive models from federated Electronic Health RecordsInternational Journal of Medical Informatics, 2018
- Real-Time Video Analytics: The Killer App for Edge ComputingComputer, 2017
- Millimeter Wave Channel Modeling and Cellular Capacity EvaluationIEEE Journal on Selected Areas in Communications, 2014
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998