Boosting the Performance of Decentralized Federated Learning via Catalyst Acceleration
Preprint
Abstract
Decentralized Federated Learning has emerged as an alternative to centralized architectures due to its faster training, privacy preservation, and reduced communication overhead. In decentralized communication, the server aggregation phase in Centralized Federated Learning shifts to the client side, which means that clients connect with each other in a peer-to-peer manner. However, compared to the centralized mode, data heterogeneity in Decentralized Federated Learning will cause larger variances between aggregated models, which leads to slow convergence in training and poor generalization performance in tests. To address these issues, we introduce Catalyst Acceleration and propose an acceleration Decentralized Federated Learning algorithm called DFedCata. It consists of two main components: the Moreau envelope function, which primarily addresses parameter inconsistencies among clients caused by data heterogeneity, and Nesterov's extrapolation step, which accelerates the aggregation phase. Theoretically, we prove the optimization error bound and generalization error bound of the algorithm, providing a further understanding of the nature of the algorithm and the theoretical perspectives on the hyperparameter choice. Empirically, we demonstrate the advantages of the proposed algorithm in both convergence speed and generalization performance on CIFAR10/100 with various non-iid data distributions. Furthermore, we also experimentally verify the theoretical properties of DFedCata.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Toward Understanding the Generalizability of Delayed Stochastic Gradient DescentIEEE Transactions on Pattern Analysis and Machine Intelligence, 2025
- Decentralized Federated Learning: A Survey and PerspectiveIEEE Internet of Things Journal, 2024
- FedGAMMA: Federated Learning With Global Sharpness-Aware MinimizationIEEE Transactions on Neural Networks and Learning Systems, 2023
- Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and ChallengesIEEE Communications Surveys & Tutorials, 2023
- Federated Learning Via Inexact ADMMIEEE Transactions on Pattern Analysis and Machine Intelligence, 2023
- FedPD: A Federated Learning Framework With Adaptivity to Non-IID DataIEEE Transactions on Signal Processing, 2021
- Privacy-Preserving Asynchronous Vertical Federated Learning Algorithms for Multiparty Collaborative LearningIEEE Transactions on Neural Networks and Learning Systems, 2021
- A Privacy-Preserving Distributed Contextual Federated Online Learning Framework with Big Data Support in Social Recommender SystemsIEEE Transactions on Knowledge and Data Engineering, 2019
- A Proximal Stochastic Gradient Method with Progressive Variance ReductionSIAM Journal on Optimization, 2014
- Anderson Acceleration for Fixed-Point IterationsSIAM Journal on Numerical Analysis, 2011