Order Optimal Bounds for One-Shot Federated Learning over non-Convex Loss Functions
Preprint
Abstract
We consider the problem of federated learning in a one-shot setting in which there are $m$ machines, each observing $n$ sample functions from an unknown distribution on non-convex loss functions. Let $F:[-1,1]^d\to\mathbb{R}$ be the expected loss function with respect to this unknown distribution. The goal is to find an estimate of the minimizer of $F$. Based on its observations, each machine generates a signal of bounded length $B$ and sends it to a server. The server collects signals of all machines and outputs an estimate of the minimizer of $F$. We show that the expected loss of any algorithm is lower bounded by $\max\big(1/(\sqrt{n}(mB)^{1/d}), 1/\sqrt{mn}\big)$, up to a logarithmic factor. We then prove that this lower bound is order optimal in $m$ and $n$ by presenting a distributed learning algorithm, called Multi-Resolution Estimator for Non-Convex loss function (MRE-NC), whose expected loss matches the lower bound for large $mn$ up to polylogarithmic factors.
Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Privacy and Robustness in Federated Learning: Attacks and DefensesIEEE Transactions on Neural Networks and Learning Systems, 2022
- An Efficient Framework for Clustered Federated LearningIEEE Transactions on Information Theory, 2022
- Communication-efficient federated learning via knowledge distillationNature Communications, 2022
- Privacy-Enhanced Federated Learning Against Poisoning AdversariesIEEE Transactions on Information Forensics and Security, 2021
- Robust and Communication-Efficient Federated Learning From Non-i.i.d. DataIEEE Transactions on Neural Networks and Learning Systems, 2019
- Differentially Private Asynchronous Federated Learning for Mobile Edge Computing in Urban InformaticsIEEE Transactions on Industrial Informatics, 2019
- Adaptive Federated Learning in Resource Constrained Edge Computing SystemsIEEE Journal on Selected Areas in Communications, 2019
- Communication-Efficient Distributed Statistical InferenceJournal of the American Statistical Association, 2018
- Approximation Theorems of Mathematical Statistics1980