TailorFL
Authors:
Open Access
- 6 November 2022
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 592-606
Abstract
Federated learning (FL) enables distributed mobile devices to collaboratively learn a shared model without exposing their raw data. However, heterogeneous devices usually have limited and different available resources, i.e., system heterogeneity, for model training and communicating, while the diverse data distribution among devices, i.e., data heterogeneity, may result in significant performance degradation. In this paper, we propose TailorFL, a dual-personalized FL framework, which tailors a submodel for each device with personalized structure for training and personalized parameters for local inference. To achieve this, we first excavate the personalization principle for data heterogeneous FL via in-depth empirical studies, and based on which, we propose a resource-aware and data-directed pruning strategy that makes each device's submodel structure match its resource capability and correlate with its local data distribution. To aggregate the submodels while preserving their dual personalization properties, we design a scaling-based aggregation strategy that scales parameters with the pruning rate of submodels and aggregates the overlapped parameters. Moreover, to further promote beneficial and restrain detrimental collaborations among devices, we propose a server-assisted model-tuning mechanism, which dynamically tunes device's submodel structure at the server side with the global view of device's data distribution similarities. Extensive experiments demonstrate that compared to the status quo approaches, TailorFL achieves an average of 22% increase in inference accuracy, and reduces the memory, computation, and communication costs for model training simultaneously.Keywords
Funding Information
- Young Elite Scientist Sponsorship Program by CAST (YESS20200238)
- Guoqiang Institute, Tsinghua University
- Natural Science Foundation of Hunan Province, China (2020JJ2050, 2021JJ20079)
- National Key R&D Program of China (2022YFF0604502)
- National Natural Science Foundation of China (62122095, 62072472, 62002389, U19A2067)
- Tsinghua University (AIR)-Asiainfo Technologies (China) Inc. Joint Research Center (20203910074)
- Xiaomi AI Innovation Research (202-422-002)
- Key Research and Development Project of Hunan Province, China (2022GK2013)
- Young Talents Plan of Hunan Province, China (2021RC3004)
This publication has 27 references indexed in Scilit:
- SAFA: A Semi-Asynchronous Protocol for Fast Federated Learning With Low OverheadIEEE Transactions on Computers, 2021
- Federated Learning in Mobile Edge Networks: A Comprehensive SurveyIEEE Communications Surveys & Tutorials, 2020
- Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive SurveyProceedings of the IEEE, 2020
- PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruning2020
- A Survey on End-Edge-Cloud Orchestrated Network Computing ParadigmsACM Computing Surveys, 2019
- Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge ComputingProceedings of the IEEE, 2019
- ADMM-NN2019
- Adaptive Federated Learning in Resource Constrained Edge Computing SystemsIEEE Journal on Selected Areas in Communications, 2019
- NestDNN2018
- Detection of traffic signs in real-world images: The German traffic sign detection benchmark2013