Communication-efficient Federated Learning with Single-Step Synthetic Features Compressor for Faster Convergence
Open Access
- 1 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 5008-5017
Abstract
Reducing communication overhead in federated learning (FL) is challenging but crucial for large-scale distributed privacy-preserving machine learning. While methods utilizing sparsification or other techniques can largely reduce the communication overhead, the convergence rate is also greatly compromised. In this paper, we propose a novel method named Single-Step Synthetic Features Compressor (3SFC) to achieve communication-efficient FL by directly constructing a tiny synthetic dataset containing synthetic features based on raw gradients. Therefore, 3SFC can achieve an extremely low compression rate when the constructed synthetic dataset contains only one data sample. Additionally, the compressing phase of 3SFC utilizes a similarity-based objective function so that it can be optimized with just one step, considerably improving its performance and robustness. To minimize the compressing error, error feedback (EF) is also incorporated into 3SFC. Experiments on multiple datasets and models suggest that 3SFC has significantly better convergence rates compared to competing methods with lower compression rates (i.e., up to 0.02%). Furthermore, ablation studies and visualizations show that 3SFC can carry more information than competing methods for every communication round, further validating its effectiveness.Keywords
All Related Versions
Funding Information
- National Science Foundation
This publication has 16 references indexed in Scilit:
- Why Batch Normalization Damage Federated Learning on Non-IID Data?IEEE Transactions on Neural Networks and Learning Systems, 2023
- Federated Learning on Non-IID Data Silos: An Experimental Study2022
- Communication-efficient federated learning via knowledge distillationNature Communications, 2022
- Communication-Efficient Federated Learning Based on Compressed SensingIEEE Internet of Things Journal, 2021
- Designing Network Design Spaces2020
- Robust and Communication-Efficient Federated Learning From Non-i.i.d. DataIEEE Transactions on Neural Networks and Learning Systems, 2019
- EMNIST: Extending MNIST to handwritten letters2017
- Deep Residual Learning for Image Recognition2016
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]IEEE Signal Processing Magazine, 2012