Data-Free Evaluation of User Contributions in Federated Learning
Authors:
- Hongtao LvHLHongtao Lv
- Zhenzhe ZhengZZZhenzhe Zheng
- Tie LuoTie Luo
- Fan WuFWFan Wu
- Shaojie TangSTShaojie Tang
- Lifeng HuaLHLifeng Hua
- Rongfei JiaRJRongfei Jia
- Chengfei LvCLChengfei Lv
- 18 October 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-8
Abstract
Federated learning (FL) trains a machine learning model on mobile devices in a distributed manner using each device’s private data and computing resources. A critical issues is to evaluate individual users’ contributions so that (1) users’ effort in model training can be compensated with proper incentives and (2) malicious and low-quality users can be detected and removed. The state-of-the-art solutions require a representative test dataset for the evaluation purpose, but such a dataset is often unavailable and hard to synthesize. In this paper, we propose a method called Pairwise Correlated Agreement (PCA) based on the idea of peer prediction to evaluate user contribution in FL without a test dataset. PCA achieves this using the statistical correlation of the model parameters uploaded by users. We then apply PCA to designing (1) a new federated learning algorithm called Fed-PCA, and (2) a new incentive mechanism that guarantees truthfulness. We evaluate the performance of PCA and Fed-PCA using the MNIST dataset and a large industrial product recommendation dataset. The results demonstrate that our Fed-PCA outperforms the canonical FedAvg algorithm and other baseline methods in accuracy, and at the same time, PCA effectively incentivizes users to behave truthfully.Keywords
All Related Versions
Funding Information
- National Science Foundation
This publication has 20 references indexed in Scilit:
- Deep Learning with Differential Privacy2016
- Informed Truthfulness in Multi-Task Peer Prediction2016
- A survey of machine learning for big data processingEURASIP Journal on Advances in Signal Processing, 2016
- Crowdsourced judgement elicitation with endogenous proficiency2013
- Computing and Analyzing the Sensitivity of MLP Due to the Errors of the i.i.d. Inputs and Weights Based on CLTIEEE Transactions on Neural Networks, 2010
- Spontaneous Dynamics of Asymmetric Random Recurrent Spiking Neural NetworksNeural Computation, 2006
- Eliciting Informative Feedback: The Peer-Prediction MethodManagement Science, 2005
- The selection of weight accuracies for MadalinesIEEE Transactions on Neural Networks, 1995
- The Proof and Measurement of Association Between Two Things.1961
- The Kolmogorov-Smirnov Test for Goodness of FitJournal of the American Statistical Association, 1951