SplitGP: Achieving Both Generalization and Personalization in Federated Learning
Preprint
Abstract
A fundamental challenge to providing edge-AI services is the need for a machine learning (ML) model that achieves personalization (i.e., to individual clients) and generalization (i.e., to unseen data) properties concurrently. Existing techniques in federated learning (FL) have encountered a steep tradeoff between these objectives and impose large computational requirements on edge devices during training and inference. In this paper, we propose SplitGP, a new split learning solution that can simultaneously capture generalization and personalization capabilities for efficient inference across resource-constrained clients (e.g., mobile/IoT devices). Our key idea is to split the full ML model into client-side and server-side components, and impose different roles to them: the client-side model is trained to have strong personalization capability optimized to each client's main task, while the server-side model is trained to have strong generalization capability for handling all clients' out-of-distribution tasks. We analytically characterize the convergence behavior of SplitGP, revealing that all client models approach stationary points asymptotically. Further, we analyze the inference time in SplitGP and provide bounds for determining model split ratios. Experimental results show that SplitGP outperforms existing baselines by wide margins in inference time and test accuracy for varying amounts of out-of-distribution samples.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- FedMes: Speeding Up Federated Learning With Multiple Edge ServersIEEE Journal on Selected Areas in Communications, 2021
- Advances and Open Problems in Federated Learning2021
- A Joint Learning and Communications Framework for Federated Learning Over Wireless NetworksIEEE Transactions on Wireless Communications, 2020
- Federated Learning: Challenges, Methods, and Future DirectionsIEEE Signal Processing Magazine, 2020
- Qsparse-Local-SGD: Distributed SGD With Quantization, Sparsification, and Local ComputationsIEEE Journal on Selected Areas in Information Theory, 2020
- Federated Learning Over Wireless Fading ChannelsIEEE Transactions on Wireless Communications, 2020
- Scheduling Policies for Federated Learning in Wireless NetworksIEEE Transactions on Communications, 2019
- Adaptive Federated Learning in Resource Constrained Edge Computing SystemsIEEE Journal on Selected Areas in Communications, 2019
- Distributed learning of deep neural network over multiple agentsJournal of Network and Computer Applications, 2018