Personalized Federated Learning with Gaussian Processes
Preprint
Abstract
Federated learning aims to learn a global model that performs well on client devices with limited cross-client communication. Personalized federated learning (PFL) further extends this setup to handle data heterogeneity between clients by learning personalized models. A key challenge in this setting is to learn effectively across clients even though each client has unique data that is often limited in size. Here we present pFedGP, a solution to PFL that is based on Gaussian processes (GPs) with deep kernel learning. GPs are highly expressive models that work well in the low data regime due to their Bayesian nature. However, applying GPs to PFL raises multiple challenges. Mainly, GPs performance depends heavily on access to a good kernel function, and learning a kernel requires a large training set. Therefore, we propose learning a shared kernel function across all clients, parameterized by a neural network, with a personal GP classifier for each client. We further extend pFedGP to include inducing points using two novel methods, the first helps to improve generalization in the low data regime and the second reduces the computational cost. We derive a PAC-Bayes generalization bound on novel clients and empirically show that it gives non-vacuous guarantees. Extensive experiments on standard PFL benchmarks with CIFAR-10, CIFAR-100, and CINIC-10, and on a new setup of learning under input noise show that pFedGP achieves well-calibrated predictions while significantly outperforming baseline methods, reaching up to 21% in accuracy gain.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- FedCor: Correlation-Based Active Client Selection Strategy for Heterogeneous Federated Learning2022
- Federated Generalized Bayesian Learning via Distributed Stein Variational Gradient DescentIEEE Transactions on Signal Processing, 2022
- FedLoc: Federated Learning Framework for Data-Driven Cooperative Localization and Location Data ProcessingIEEE Open Journal of Signal Processing, 2020
- Federated Learning: Challenges, Methods, and Future DirectionsIEEE Signal Processing Magazine, 2020
- PAC-Bayesian Generalisation Error Bounds for Gaussian Process ClassificationJournal of Machine Learning Research : Jmlr, 2003
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998