FedRSL: Representation Subspace Learning in Model-Heterogeneous Federated Learning
- 6 February 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 36 (6) , 9010-9022
Abstract
Model-heterogeneous federated learning (MHFL), supporting FL collaboration across clients with heterogeneous models, has become a more practical FL paradigm. Existing MHFL methods enable knowledge fusion over heterogeneous client models by sharing partial homogeneous parameters or extracted label-wise average representations, suffering from model performance bottlenecks and privacy leakage risks. To bridge this gap, we propose a novel model-heterogeneous Federated learning method with homogeneous Representation Subspace Learning (FedRSL) instead of sharing model parameters or representations. In FedRSL, each client’s local heterogeneous model comprises a feature extractor and a prediction header. (1) We construct a homogeneous representation subspace for each client to learn local representation knowledge, and the server aggregates them to generate the global representation subspace for representation knowledge fusion. (2) To facilitate representation learning capability while maintaining efficient communication and computation, we design a lightweight linear model as the homogeneous low-rank linear representation subspace. For each local data sample, its personalized representation extracted by the feature extractor is processed by the global representation subspace to produce the corresponding generalized representation. (3) To effectively bi-transfer global generalized and local personalized knowledge, we reduce the distance between the local personalized representation and the corresponding generalized representation. Experiments on 3 computer vision and 1 natural language processing benchmark datasets over 6 baselines demonstrate that FedRSL obtains state-of-the-art model accuracy (up to 5.51% accuracy improvement) while consuming low communication and computation overheads.Keywords
Funding Information
- National Natural Science Foundation of China (62406219)
- National Natural Science Foundation of China (U23B2049)
- National Natural Science Foundation of China (62376194)
- National Natural Science Foundation of China (62436001)
- National Natural Science Foundation of China (62272252)
- National Natural Science Foundation of China (62272253)
- National Natural Science Foundation of China
This publication has 42 references indexed in Scilit:
- pFedKT: Personalized federated learning with dual knowledge transferKnowledge-Based Systems, 2024
- FedDIP: Federated Learning with Extreme Dynamic Pruning and Incremental Regularization2023
- Decentralized federated learning through proxy model sharingNature Communications, 2023
- Towards Data-Independent Knowledge Transfer in Model-Heterogeneous Federated LearningIEEE Transactions on Computers, 2023
- FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Models2022
- Learn from Others and Be Yourself in Heterogeneous Federated Learning2022
- Communication-efficient federated learning via knowledge distillationNature Communications, 2022
- Heterogeneous Model Fusion Federated Learning Mechanism Based on Model MappingIEEE Internet of Things Journal, 2021
- Cooperative Learning VIA Federated Distillation OVER Fading Channels2020
- Wireless Federated Distillation for Distributed Edge Learning with Heterogeneous Data2019