Elastic Aggregation for Federated Optimization
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 12187-12197
Abstract
Federated learning enables the privacy-preserving training of neural network models using real-world data across distributed clients. FedAvg has become the preferred optimizer for federated learning because of its simplicity and effectiveness. FedAvg uses naïve aggregation to update the server model, interpolating client models based on the number of instances used in their training. However, naïve aggregation suffers from client drift when the data is heterogenous (non-IID), leading to unstable and slow convergence. In this work, we propose a novel aggregation approach, elastic aggregation, to overcome these issues. Elastic aggregation interpolates client models adaptively according to parameter sensitivity, which is measured by computing how much the overall prediction function output changes when each parameter is changed. This measurement is performed in an unsupervised and online manner. Elastic aggregation reduces the magnitudes of updates to the more sensitive parameters so as to prevent the server model from drifting to any one client distribution, and conversely boosts updates to the less sensitive parameters to better explore different client distributions. Empirical results on real and synthetic data as well as analytical results show that elastic aggregation leads to efficient training in both convex and nonconvex settings while being fully agnostic to client heterogeneity and robust to large numbers of clients, partial participation, and imbalanced data. Finally, elastic aggregation works well with other federated optimizers and achieves significant improvements across the board.Keywords
Funding Information
- NSFC (62072449)
This publication has 12 references indexed in Scilit:
- FedDANE: A Federated Newton-Type Method2019
- Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep LearningProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Adaptive Federated Learning in Resource Constrained Edge Computing SystemsIEEE Journal on Selected Areas in Communications, 2019
- Federated Learning of N-Gram Language Models2019
- Federated Learning for Ultra-Reliable Low-Latency V2V Communications2018
- Federated learning of predictive models from federated Electronic Health RecordsInternational Journal of Medical Informatics, 2018
- Practical Secure Aggregation for Privacy-Preserving Machine Learning2017
- Convergence Rate of Distributed ADMM Over NetworksIEEE Transactions on Automatic Control, 2017
- Deep Residual Learning for Image Recognition2016
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998