SAFARI: Sparsity-Enabled Federated Learning With Limited and Unreliable Communications
- 18 July 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Mobile Computing
- Vol. 23 (5) , 4819-4831
Abstract
Federated learning (FL) enables edge devices to collaboratively learn a model in a distributed fashion. Many existing researches have focused on improving communication efficiency of high-dimensional models and addressing bias caused by local updates. However, most FL algorithms are either based on reliable communications or assuming fixed and known unreliability characteristics. In practice, networks could suffer from dynamic channel conditions and non-deterministic disruptions, with time-varying and unknown characteristics. To this end, in this paper we propose a sparsity-enabled FL framework with both improved communication efficiency and bias reduction, termed as SAFARI. It makes use of similarity among client models to rectify and compensate for bias that results from unreliable communications. More precisely, sparse learning is implemented on local clients to mitigate communication overhead, while to cope with unreliable communications, a similarity-based compensation method is proposed to provide surrogates for missing model updates. With respect to sparse models, we analyze SAFARI under bounded dissimilarity. It is demonstrated that SAFARI under unreliable communications is guaranteed to converge at the same rate as the standard FedAvg with perfect communications. Implementations and evaluations on the CIFAR-10 dataset validate the effectiveness of SAFARI by showing that it can achieve the same convergence speed and accuracy as FedAvg with perfect communications, with up to 60% of the model weights being pruned and a high percentage of client updates missing in each round of model updates.Keywords
All Related Versions
Funding Information
- National Key R&D Program of China (2022ZD0160504)
- Tsinghua Shenzhen International Graduate School-Shenzhen Pengrui Young Faculty Program of Shenzhen Pengrui Foundation (SZPR2023005)
- Tsinghua-Toyota Joint Research Institute Inter-disciplinary Program
- Tsinghua University (20203910074)
This publication has 11 references indexed in Scilit:
- Decentralized Federated Learning With Unreliable CommunicationsIEEE Journal of Selected Topics in Signal Processing, 2022
- Federated Learning and Wireless CommunicationsIEEE Wireless Communications, 2021
- Adaptive Quantization of Model Updates for Communication-Efficient Federated Learning2021
- A Secure Federated Transfer Learning FrameworkIEEE Intelligent Systems, 2020
- Robust and Communication-Efficient Federated Learning From Non-i.i.d. DataIEEE Transactions on Neural Networks and Learning Systems, 2019
- Federated Machine LearningACM Transactions on Intelligent Systems and Technology, 2019
- Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network scienceNature Communications, 2018
- Practical Secure Aggregation for Privacy-Preserving Machine Learning2017
- Deep Learning with Differential Privacy2016
- Deep Residual Learning for Image Recognition2016