FedDIP: Federated Learning with Extreme Dynamic Pruning and Incremental Regularization
Authors:
- 1 December 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1187-1192
Abstract
Federated Learning (FL) has been successfully adopted for distributed training and inference of large-scale Deep Neural Networks (DNNs). However, DNNs are characterized by an extremely large number of parameters, thus, yielding significant challenges in exchanging these parameters among distributed nodes and managing the memory. Although recent DNN compression methods (e.g., sparsification, pruning) tackle such challenges, they do not holistically consider an adaptively controlled reduction of parameter exchange while maintaining high accuracy levels. We, therefore, contribute with a novel FL framework (coined FedDIP), which combines (i) dynamic model pruning with error feedback to eliminate redundant information exchange, which contributes to significant performance improvement, with (ii) incremental regularization that can achieve extreme sparsity of models. We provide convergence analysis of FedDIP and report on a comprehensive performance and comparative assessment against state-of-the-art methods using benchmark data sets and DNN models. Our results showcase that FedDIP not only controls the model sparsity but efficiently achieves similar or better performance compared to other model pruning methods adopting incremental regularization during distributed model training. The code is available at: https://github.com/EricLoong/feddip.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China
This publication has 15 references indexed in Scilit:
- Federated Learning for Wireless Communications: Motivation, Opportunities, and ChallengesIEEE Communications Magazine, 2020
- Secure, privacy-preserving and federated machine learning in medical imagingNature Machine Intelligence, 2020
- Federated Learning: Challenges, Methods, and Future DirectionsIEEE Signal Processing Magazine, 2020
- Federated LearningSynthesis Lectures on Artificial Intelligence and Machine Learning, 2019
- PruneTrain2019
- Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge ComputingProceedings of the IEEE, 2019
- A Systematic DNN Weight Pruning Framework Using Alternating Direction Method of Multipliers2018
- Channel Pruning for Accelerating Very Deep Neural Networks2017
- Sparse Communication for Distributed Gradient Descent2017
- Deep Residual Learning for Image Recognition2016