FedDIP: Federated Learning with Extreme Dynamic Pruning and Incremental Regularization
Preprint
Abstract
Federated Learning (FL) has been successfully adopted for distributed training and inference of large-scale Deep Neural Networks (DNNs). However, DNNs are characterized by an extremely large number of parameters, thus, yielding significant challenges in exchanging these parameters among distributed nodes and managing the memory. Although recent DNN compression methods (e.g., sparsification, pruning) tackle such challenges, they do not holistically consider an adaptively controlled reduction of parameter exchange while maintaining high accuracy levels. We, therefore, contribute with a novel FL framework (coined FedDIP), which combines (i) dynamic model pruning with error feedback to eliminate redundant information exchange, which contributes to significant performance improvement, with (ii) incremental regularization that can achieve \textit{extreme} sparsity of models. We provide convergence analysis of FedDIP and report on a comprehensive performance and comparative assessment against state-of-the-art methods using benchmark data sets and DNN models. Our results showcase that FedDIP not only controls the model sparsity but efficiently achieves similar or better performance compared to other model pruning methods adopting incremental regularization during distributed model training. The code is available at: https://github.com/EricLoong/feddip.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Convergence Analysis and System Design for Federated Learning Over Wireless NetworksIEEE Journal on Selected Areas in Communications, 2021
- Federated Learning for Open Banking2020
- Federated Learning for Healthcare InformaticsJournal of Healthcare Informatics Research, 2020
- Federated Learning for Wireless Communications: Motivation, Opportunities, and ChallengesIEEE Communications Magazine, 2020
- Secure, privacy-preserving and federated machine learning in medical imagingNature Machine Intelligence, 2020
- Federated Learning: Challenges, Methods, and Future DirectionsIEEE Signal Processing Magazine, 2020
- Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge ComputingProceedings of the IEEE, 2019
- A Systematic DNN Weight Pruning Framework Using Alternating Direction Method of Multipliers2018
- Channel Pruning for Accelerating Very Deep Neural Networks2017
- Deep Residual Learning for Image Recognition2016