Order-Optimal Byzantine-Robust Learning Under Heterogeneity via Fair Gradient Clipping
- 11 November 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cybernetics
- Vol. 56 (3) , 1503-1514
Abstract
Byzantine-robust distributed or federated learning (FL) refers to providing reliable performance under Byzantine attacks, which violate the prescribed protocols and transmit arbitrary information to the server to hamper the convergence of machine learning (ML) algorithms, via designing resilient aggregation rules to combat attacks. Although numerous robust rules have been suggested, their performance degrades for heterogeneous data. A few techniques have been exploited to handle this problem, but they either require preaggregation operations, hence increasing the computational load, or lack breakdown point analysis of their rules. This article proposes a new aggregation rule, which clips the gradients received from all workers according to the distance between the gradient and the aggregation center. That is, when the distance is larger than the radius $\gamma $ , the gradient will be clipped, and the longer the distance, the closer the clipped gradient is to the center. We theoretically analyze that the breakdown point of the developed rule is 0.5, the maximum value for robust aggregators. Moreover, our rule achieves order-optimal Byzantine-robust training error under data heterogeneity, while the median-based schemes, such as coordinate-wise median (CM) and geometric median (GM), are suboptimal. Experimental results demonstrate that the devised aggregation mechanism can handle different attacks well and outperforms the existing rules.Keywords
Funding Information
- Research Grant of Shenzhen Research Institute, City University of Hong Kong, Shenzhen, China (R-IND25501)
This publication has 27 references indexed in Scilit:
- Robust low-rank matrix completion via sparsity-inducing regularizerSignal Processing, 2024
- Selective Trimmed Average: A Resilient Federated Learning Algorithm With Deterministic Guarantees on the Optimality ApproximationIEEE Transactions on Cybernetics, 2024
- FedLGA: Toward System-Heterogeneity of Federated Learning via Local Gradient ApproximationIEEE Transactions on Cybernetics, 2023
- Robust Matrix Completion Based on Factorization and Truncated-Quadratic Loss FunctionIEEE Transactions on Circuits and Systems for Video Technology, 2022
- Coarse to Fine Two-Stage Approach to Robust Tensor Completion of Visual DataIEEE Transactions on Cybernetics, 2022
- Byzantine-Resilient SGD in High Dimensions on Heterogeneous Data2021
- Federated Continuous Learning With Broad Network ArchitectureIEEE Transactions on Cybernetics, 2021
- Robust Statistics for Signal Processing2018
- Deep Residual Learning for Image Recognition2016
- Wiley Series in Probability and Statistics2008