Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Models
Top Cited Papers
- 31 December 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Mobile Computing
- Vol. 25 (6) , 8782-8797
Abstract
Recently, there has been a surge in the development of advanced intelligent generative content (AIGC), especially large language models (LLMs). For many downstream tasks, it is necessary to fine-tune LLMs using private data. While federated learning offers a promising privacy-preserving solution to LLM fine-tuning, the substantial size of an LLM, combined with high computational and communication demands, makes it hard to apply to handle downstream tasks. More importantly, private edge servers often possess varying computing and network resources in real-world scenarios, introducing additional complexities to LLM fine-tuning. To tackle these problems, we design and implement an automated federated pipeline, named FedPipe, to fine-tune LLMs on heterogeneous edge servers with minimal training cost and no additional inference latency. FedPipe firstly identifies the weights to be fine-tuned based on their contributions to the LLM training. It then configures a low-rank adapter for each selected weight within the resource constraints of the edge server and aggregates these local adapters from all edge servers to fine-tune the whole LLM. Finally, it appropriately quantizes the parameters of LLM to reduce memory consumption according to the requirements of edge servers. Extensive experiments demonstrate that FedPipe expedites model training and achieves higher accuracy than the state-of-the-art benchmarks.Keywords
All Related Versions
Funding Information
- Natural Science Foundation of Shanghai Municipality (25ZR1401016)
- The Hong Kong Jockey Club Charities Trust
- Hong Kong SAR government
- Research Grants Council of Hong Kong (27213824)
- CRS (HKU702/24)
This publication has 38 references indexed in Scilit:
- GPT-FL: Generative Pre-Trained Model-Assisted Federated Learning2025
- Mobile Edge Intelligence for Large Language Models: A Contemporary SurveyIEEE Communications Surveys & Tutorials, 2025
- Federated Prompting and Chain-of-Thought Reasoning for Improving LLMs Answering2023
- Federated benchmarking of medical artificial intelligence with MedPerfNature Machine Intelligence, 2023
- TimelyFL: Heterogeneity-aware Asynchronous Federated Learning with Adaptive Partial Training2023
- Online Federated Learning based Object Detection across Autonomous Vehicles in a Virtual World2023
- BalanceFL: Addressing Class Imbalance in Long-Tail Federated Learning2022
- Device Sampling for Heterogeneous Federated Learning: Theory, Algorithms, and Implementation2021
- Importance Estimation for Neural Network Pruning2019
- Federated Learning over Wireless Networks: Optimization Model Design and Analysis2019