FedPAGE: Pruning Adaptively Toward Global Efficiency of Heterogeneous Federated Learning
- 12 December 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Networking
- Vol. 32 (3) , 1873-1887
Abstract
When workers are heterogeneous in computing and transmission capabilities, the global efficiency of federated learning suffers from the straggler issue, i.e., the slowest worker drags down the overall training process. We propose a novel and efficient federated learning framework named FedPAGE, where workers perform distributed pruning adaptively towards global efficiency, i.e., fast training and high accuracy. For fast training, we develop a pruning rate learning approach generating an adaptive pruning rate for each worker, making the overall update time approximate to the fastest worker’s update time, i.e., no stragglers. For high accuracy, we find that structural similarity between sub-models is essential to global model accuracy in the distributed pruning, and thus propose the CIG_X pruning scheme to ensure maximum similarity. Meanwhile, we adopt the sparse training and design model aggregating of different size sub-models to cope with distributed pruning. We prove the convergence of FedPAGE and demonstrate the effectiveness of FedPAGE on image classification and natural language inference tasks. Compared with the state-of-the-art, FedPAGE achieves higher accuracy with the same speedup ratio.Keywords
Funding Information
- National Natural Science Foundation of China (U22B2031, 61932016, 62132011, 62202258)
- National Science Foundation for Distinguished Young Scholars of China (61825204)
- Beijing Outstanding Young Scientist Program (BJJWZYJH01201910003011)
- China Postdoctoral Science Foundation (2021M701894)
- China National Postdoctoral Program for Innovative Talents, Shuimu Tsinghua Scholar Program
- Ant Group through the CCF-Ant Innovative Research Program (CCF-AFSGRF20210023)
This publication has 30 references indexed in Scilit:
- Towards Efficient Model Compression via Learned Global Ranking2020
- SecCL: Securing Collaborative Learning Systems via Trusted Bulletin BoardsIEEE Communications Magazine, 2020
- PruneTrain2019
- Robust and Communication-Efficient Federated Learning From Non-i.i.d. DataIEEE Transactions on Neural Networks and Learning Systems, 2019
- Towards Optimal Structured CNN Pruning via Generative Adversarial Learning2019
- Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration2019
- Adaptive Federated Learning in Resource Constrained Edge Computing SystemsIEEE Journal on Selected Areas in Communications, 2019
- Practical Secure Aggregation for Privacy-Preserving Machine Learning2017
- Learning Efficient Convolutional Networks through Network Slimming2017
- Deep Residual Learning for Image Recognition2016