Abstract
Federated learning (FL) enables collaborative training on decentralized data while preserving the data owners’ privacy, under the orchestration of a central server. FL has seen tremendous growth and advancements in recent years. Despite its progress, FL faces a significant challenge raised by data heterogeneity, leading to a slower convergence rate and a larger performance gap compared to centralized training. In this work, we empirically reveal that direct applying empirical risk minimizing (ERM) on skewed client training data causes the client model suffers from biased predictions towards majority classes. To address this problem, we propose a model agnostic instance reweighing method (MAIR). At a coarse-grained level, MAIR adjusts the logits predictions for each class to counteract the data heterogeneity. At a fine-grained level, it dynamically reweighs the importance of individual training samples with a predictive meta network. As a results, MAIR prevents client models from over-fitting on heterogeneous data and therefore substantially reduces client drift. Theoretically, we justify its non-convex convergence property. Extensive experiments demonstrate that MAIR reliably speeds up convergence and improves the quality of global models, outperforming its best competitor by a clear margin. It notably delivers $8.3\%$ improvements on ImageNet subset and achieves $67.6\%$ energy footprint reduction on CIFAR-100 over the FedAvg baseline. Our findings also suggest that improving the performance of FL-trained models necessitates rethinking clients’ local optimization objectives, and ERM should thus no longer be viewed as a de facto standard in FL under data heterogeneity.
Funding Information
  • Technology Development Fund of Macao S.A.R (0123/2022/AFJ)
  • Technology Development Fund of Macao S.A.R (0081/2022/A2)
  • National Natural Science Foundation of China (62376263)
  • Basic and Applied Basic Research Foundation of Guangdong Province (2023B1515130002)
  • Natural Science Foundation of Guangdong Province (2024A1515030209)
  • Science, Technology and Innovation Commission of Shenzhen Municipality (JCYJ20230807140507015)
  • SICC
  • SKL-IOTSC
  • Universidade de Macau