Concept Matching: Clustering-based Federated Continual Learning
Preprint
Abstract
Federated Continual Learning (FCL) has emerged as a promising paradigm that combines Federated Learning (FL) and Continual Learning (CL). To achieve good model accuracy, FCL needs to tackle catastrophic forgetting due to concept drift over time in CL, and to overcome the potential interference among clients in FL. We propose Concept Matching (CM), a clustering-based framework for FCL to address these challenges. The CM framework groups the client models into concept model clusters, and then builds different global models to capture different concepts in FL over time. In each round, the server sends the global concept models to the clients. To avoid catastrophic forgetting, each client selects the concept model best-matching the concept of the current data for further fine-tuning. To avoid interference among client models with different concepts, the server clusters the models representing the same concept, aggregates the model weights in each cluster, and updates the global concept model with the cluster model of the same concept. Since the server does not know the concepts captured by the aggregated cluster models, we propose a novel server concept matching algorithm that effectively updates a global concept model with a matching cluster model. The CM framework provides flexibility to use different clustering, aggregation, and concept matching algorithms. The evaluation demonstrates that CM outperforms state-of-the-art systems and scales well with the number of clients and the model size.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- ClusterFL: A Clustering-based Federated Learning System for Human Activity RecognitionACM Transactions on Sensor Networks, 2022
- FLSys: Toward an Open Ecosystem for Federated Learning Mobile AppsIEEE Transactions on Mobile Computing, 2022
- An Efficient Framework for Clustered Federated LearningIEEE Transactions on Information Theory, 2022
- Online continual learning in image classification: An empirical surveyNeurocomputing, 2022
- SAFA: A Semi-Asynchronous Protocol for Fast Federated Learning With Low OverheadIEEE Transactions on Computers, 2021
- A continual learning survey: Defying forgetting in classification tasksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2021
- Federated Learning With Differential Privacy: Algorithms and Performance AnalysisIEEE Transactions on Information Forensics and Security, 2020
- Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive SurveyProceedings of the IEEE, 2020
- Continuous learning in single-incremental-task scenariosNeural Networks, 2019
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998