AugFL: Augmenting Federated Learning With Pretrained Models
- 15 April 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Networking
- Vol. 33 (4) , 1870-1885
Abstract
Federated Learning (FL) has garnered widespread interest in recent years. However, owing to strict privacy policies or limited storage capacities of training participants such as IoT devices, its effective deployment is often impeded by the scarcity of training data in practical decentralized learning environments. In this paper, we study enhancing FL with the aid of (large) pre-trained models (PMs), that encapsulate wealthy general/domain-agnostic knowledge, to alleviate the data requirement in conducting FL from scratch. Specifically, we consider a networked FL system formed by a central server and distributed clients. First, we formulate the PM-aided personalized FL as a regularization-based federated meta-learning problem, where clients join forces to learn a meta-model with knowledge transferred from a private PM stored at the server. Then, we develop an inexact-ADMM-based algorithm, AugFL, to optimize the problem with no need to expose the PM or incur additional computational costs to local clients. Further, we establish theoretical guarantees for AugFL in terms of communication complexity, adaptation performance, and the benefit of knowledge transfer in general non-convex cases. Extensive experiments corroborate the efficacy and superiority of AugFL over existing baselines.Keywords
All Related Versions
Funding Information
- National Key Research and Development Program of China (2022YFB2703303)
- National Natural Science Foundation of China (62432004, 62411540034, 62402267)
- Guoqiang Institute
This publication has 42 references indexed in Scilit:
- Decoupled Knowledge Distillation2022
- An Asynchronous Quasi‐Cloud/Edge/Client Collaborative Federated Learning Mechanism for Fault DiagnosisChinese Journal of Electronics, 2021
- Federated Learning for Internet of Things: A Comprehensive SurveyIEEE Communications Surveys & Tutorials, 2021
- Pre-trained models: Past, present and futureAI Open, 2021
- Mitigating Information Leakage in Image Representations: A Maximum Entropy Approach2019
- Relational Knowledge Distillation2019
- Structured nonconvex and nonsmooth optimization: algorithms and iteration complexity analysisComputational Optimization and Applications, 2018
- Convergence of multi-block Bregman ADMM for nonconvex composite problemsScience China Information Sciences, 2018
- Global Convergence of ADMM in Nonconvex Nonsmooth OptimizationJournal of Scientific Computing, 2018
- Nonconvex nonsmooth optimization via convex–nonconvex majorization–minimizationNumerische Mathematik, 2016