FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning
Open Access
- 1 December 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 34 (9) , 5531-5543
Abstract
Federated distillation (FD) is a popular novel algorithmic paradigm for Federated learning (FL), which achieves training performance competitive to prior parameter averaging-based methods, while additionally allowing the clients to train different model architectures, by distilling the client predictions on an unlabeled auxiliary set of data into a student model. In this work, we propose FEDAUX, an extension to FD, which, under the same set of assumptions, drastically improves the performance by deriving maximum utility from the unlabeled auxiliary data. FEDAUX modifies the FD training procedure in two ways: First, unsupervised pre-training on the auxiliary data is performed to find a suitable model initialization for the distributed training. Second, ε, δ)-differentially private certainty scoring is used to weight the ensemble predictions on the auxiliary data according to the certainty of each client model. Experiments on large-scale convolutional neural networks (CNNs) and transformer models demonstrate that our proposed method achieves remarkable performance improvements over state-of-the-art FL methods, without adding appreciable computation, communication, or privacy cost. For instance, when training ResNet8 on non-independent identically distributed (i.i.d.) subsets of CIFAR10, FEDAUX raises the maximum achieved validation accuracy from 30.4% to 78.1%, further closing the gap to centralized training performance. Code is available at https://github.com/fedl-repo/fedaux.Keywords
All Related Versions
Funding Information
- German Federal Ministry of Education and Research (BMBF) through the Berlin Institute for the Foundations of Learning and Data (01IS18025A, 01IS18037I)
- EU’s Horizon 2020 Project COPA EUROPE (957059)
This publication has 31 references indexed in Scilit:
- Deep Learning with Differential Privacy2016
- Deep Residual Learning for Image Recognition2016
- Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books2015
- Twenty Years of Mixture of ExpertsIEEE Transactions on Neural Networks and Learning Systems, 2012
- Mixture of experts: a literature surveyArtificial Intelligence Review, 2012
- ImageNet: A large-scale hierarchical image database2009
- Dynamically weighted ensemble neural networks for classification2002
- Popular Ensemble Methods: An Empirical StudyJournal of Artificial Intelligence Research, 1999
- On Combining Artificial Neural NetsConnection Science, 1996
- On the limited memory BFGS method for large scale optimizationMathematical Programming, 1989