Federated Learning for Extreme Label Noise: Enhanced Knowledge Distillation and Particle Swarm Optimization
Open Access
- 17 January 2025
- journal article
- research article
- Published by MDPI AG in Electronics
- Vol. 14 (2) , 366
Abstract
Federated learning, with its unique privacy protection mechanisms and distributed model training capabilities, provides an effective solution for data security by addressing the challenges associated with the inability to directly share private data due to privacy concerns. It exhibits broad application potential across various fields, particularly in scenarios such as autonomous vehicular networks, where collaborative learning is required from data sources distributed across different clients, thus optimizing and enhancing model performance. Nevertheless, in complex real-world environments, challenges such as data poisoning and labeling errors may cause some clients to introduce label noise that significantly exceeds ordinary levels, severely impacting model performance. The following conclusions are drawn from research on extreme label noise: highly polluted data severely affect the generalization capability of the global model and the stability of the training process, while the reweighting strategy can improve model performance. Based on these research conclusions, we propose a method named Enhanced Knowledge Distillation and Particle Swarm Optimization for Federated Learning (FedDPSO) to deal with extreme label noise. In FedDPSO, the server dynamically identifies extremely noisy clients based on uncertainty. It then uses the particle swarm optimization algorithm to determine client model weights for global model aggregation. In subsequent rounds, the identified extremely noisy clients construct an interpolation loss combining pseudo-label loss and knowledge distillation loss, effectively mitigating the negative impact of label noise overfitting on the local model. We carried out experiments on the CIFAR10/100 datasets to validate the effectiveness of FedDPSO. At the highest noise ratio under Beta = (0.1, 0.1), experiments show that FedDPSO improves the average accuracy on CIFAR10 by 15% compared to FedAvg and by 11% compared to the more powerful FOCUS. On CIFAR100, it outperforms FedAvg by 8% and FOCUS by 5%.Keywords
Funding Information
- Ministry of Education of Humanities and Social Science Project (19YJAZH047)
This publication has 38 references indexed in Scilit:
- Small models, big impact: A review on the power of lightweight Federated LearningFuture Generation Computer Systems, 2024
- Human memory optimization algorithm: A memory-inspired optimizer for global optimization problemsExpert Systems with Applications, 2023
- Boosted federated learning based on improved Particle Swarm Optimization for healthcare IoT devicesComputers in Biology and Medicine, 2023
- Manta ray foraging optimization based on mechanics game and progressive learning for multiple optimization problemsApplied Soft Computing, 2023
- A particle swarm optimization algorithm with empirical balance strategyChaos, Solitons & Fractals: X, 2022
- Knowledge Distillation: A SurveyInternational Journal of Computer Vision, 2021
- Dealing with Label Quality Disparity in Federated Learning2020
- Classification with label noise: a Markov chain sampling frameworkData Mining and Knowledge Discovery, 2018
- Empirical Risk Minimization2017
- Particle swarm optimization algorithm: an overviewSoft Computing, 2017