FedSSC: Shared Supervised-Contrastive Federated Learning
Preprint
Abstract
Federated learning is widely used to perform decentralized training of a global model on multiple devices while preserving the data privacy of each device. However, it suffers from heterogeneous local data on each training device which increases the difficulty to reach the same level of accuracy as the centralized training. Supervised Contrastive Learning which outperform cross-entropy tries to minimizes the difference between feature space of points belongs to the same class and pushes away points from different classes. We propose Supervised Contrastive Federated Learning in which devices can share the learned class-wise feature spaces with each other and add the supervised-contrastive learning loss as a regularization term to foster the feature space learning. The loss tries to minimize the cosine similarity distance between the feature map and the averaged feature map from another device in the same class and maximizes the distance between the feature map and that in a different class. This new regularization term when added on top of the moon regularization term is found to outperform the other state-of-the-art regularization terms in solving the heterogeneous data distribution problem.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- FedProc: Prototypical contrastive federated learning on non-IID dataFuture Generation Computer Systems, 2023
- SphereFed: Hyperspherical Federated Learning2022
- Anomalous layer-dependent lubrication on graphene-covered substrate: Competition between adhesion and plasticityApplied Surface Science, 2022
- Exploring Simple Siamese Representation Learning2021
- Momentum Contrast for Unsupervised Visual Representation Learning2020
- Federated Learning: Challenges, Methods, and Future DirectionsIEEE Signal Processing Magazine, 2020