Federated Learning with Position-Aware Neurons
Authors:
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 10072-10081
Abstract
Federated Learning (FL) fuses collaborative models from local nodes without centralizing users' data. The permutation invariance property of neural networks and the non-i.i.d. data across clients make the locally updated parameters imprecisely aligned, disabling the coordinate-based parameter averaging. Traditional neurons do not explicitly consider position information. Hence, we propose Position-Aware Neurons (PANs) as an alternative, fusing position-related values (i.e., position encodings) into neuron outputs. PANs couple themselves to their positions and minimize the possibility of dislocation, even updating on heterogeneous data. We turn on/off PANs to disable/enable the permutation invariance property of neural networks. PANs are tightly coupled with positions when applied to FL, making parameters across clients pre-aligned and facilitating coordinate-based parameter averaging. PANs are algorithm-agnostic and could universally improve existing FL algorithms. Furthermore, “FL with PANs” is simple to implement and computationally friendly.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (41901270)
- Natural Science Foundation of Jiangsu Province (BK20190296)
This publication has 17 references indexed in Scilit:
- Group NormalizationInternational Journal of Computer Vision, 2019
- Federated Machine LearningACM Transactions on Intelligent Systems and Technology, 2019
- Self-Attention with Relative Position Representations2018
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- Deep Residual Learning for Image Recognition2016
- Scaling Distributed Machine Learning with the Parameter Server2014
- The German Traffic Sign Recognition Benchmark: A multi-class classification competition2011
- Barycenters in the Wasserstein SpaceSIAM Journal on Mathematical Analysis, 2011
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- The Hungarian method for the assignment problemNaval Research Logistics Quarterly, 1955