Sign-Based Gradient Descent With Heterogeneous Data: Convergence and Byzantine Resilience
- 12 January 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 36 (2) , 3834-3846
Abstract
Communication overhead has become one of the major bottlenecks in the distributed training of modern deep neural networks. With such consideration, various quantization-based stochastic gradient descent (SGD) solvers have been proposed and widely adopted, among which signSGD with majority vote shows a promising direction because of its communication efficiency and robustness against Byzantine attackers. However, signSGD fails to converge in the presence of data heterogeneity, which is commonly observed in the emerging federated learning (FL) paradigm. In this article, a sufficient condition for the convergence of the sign-based gradient descent method is derived, based on which a novel magnitude-driven stochastic-sign-based gradient compressor is proposed to address the non-convergence issue of signSGD. The convergence of the proposed method is established in the presence of arbitrary data heterogeneity. The Byzantine resilience of sign-based gradient descent methods is quantified, and the error-feedback mechanism is further incorporated to boost the learning performance Experimental results on the MNIST dataset, the CIFAR-10 dataset, and the Tiny-ImageNet dataset corroborate the effectiveness of the proposed methods.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62301487)
- Zhejiang Provincial Natural Science Foundation of China (LQ23F010021)
- Ng Teng Fong Charitable Foundation in the form of ZJU-SUTD IDEA (188170-11102)
- Army Research Office (W911NF1810295, W911NF2210010)
- NSF (IIS-1909644, IIS-1822477, CMMI-2024688, IUSE-201345)
- U.S. National Science Foundation (ECCS-2203214)
This publication has 18 references indexed in Scilit:
- Byzantine-Robust Distributed Learning With CompressionIEEE Transactions on Signal and Information Processing over Networks, 2023
- Byzantine Tolerant Algorithms for Federated LearningIEEE Transactions on Network Science and Engineering, 2023
- Federated Learning via Plurality VoteIEEE Transactions on Neural Networks and Learning Systems, 2022
- Byzantine-robust Federated Learning through Collaborative Malicious Gradient Filtering2022
- Model Compression for Communication Efficient Federated LearningIEEE Transactions on Neural Networks and Learning Systems, 2021
- Advances and Open Problems in Federated LearningFoundations and Trends in Machine Learning, 2021
- FLTrust: Byzantine-robust Federated Learning via Trust Bootstrapping2021
- Ternary Compression for Communication-Efficient Federated LearningIEEE Transactions on Neural Networks and Learning Systems, 2020
- Robust and Communication-Efficient Federated Learning From Non-i.i.d. DataIEEE Transactions on Neural Networks and Learning Systems, 2019
- Stochastic Spectral Descent for Discrete Graphical ModelsIEEE Journal of Selected Topics in Signal Processing, 2015