Enabling Communication-Efficient Federated Learning via Distributed Compressed Sensing
Authors:
- 17 May 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-10
Abstract
Federated learning (FL) trains a shared global model by periodically aggregating gradients from local devices. Communication overhead becomes a principal bottleneck in FL since participating devices usually suffer from limited bandwidth and unreliable connections in uplink transmission. To address this problem, the gradient compression methods based on compressed sensing (CS) theory have been put forward recently. However, most existing CS-based works compress gradients independently, ignoring the gradient correlations between participants or adjacent communication rounds, which constrains the achievement of higher compression rates. In view of the above observation, we propose a novel gradient compression scheme named FedDCS, guided by distributed compressed sensing (DCS) theory. Following the design philosophy of separate encoding and joint decoding in DCS, FedDCS compresses gradients for participants in each round separately while reconstructing them at the central server jointly via fully exploiting correlated gradients from the previous round, which are known as side information (SI). Benefiting from this design, reconstruction performance is significantly improved with fewer decoding errors also iterations under the identical compression rate, and the total uploading bits to achieve model convergence are considerably reduced. Theoretical analysis and extensive experiments conducted on MNIST and Fashion-MNIST both verify the effectiveness of our approach.Keywords
Funding Information
- National Natural Science Foundation of China
This publication has 24 references indexed in Scilit:
- Robust and Communication-Efficient Federated Learning From Non-i.i.d. DataIEEE Transactions on Neural Networks and Learning Systems, 2019
- Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks2018
- Hard Thresholding Pursuit: An Algorithm for Compressive SensingSIAM Journal on Numerical Analysis, 2011
- Compressed Sensing With Quantized MeasurementsIEEE Signal Processing Letters, 2009
- Distributed compressive video sensing2009
- Bayesian Compressive SensingIEEE Transactions on Signal Processing, 2008
- An Introduction To Compressive SamplingIEEE Signal Processing Magazine, 2008
- Compressed sensingIEEE Transactions on Information Theory, 2006
- Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency informationIEEE Transactions on Information Theory, 2006
- Noiseless coding of correlated information sourcesIEEE Transactions on Information Theory, 1973