Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization
Preprint
Abstract
Federated learning (FL) is a promising paradigm to enable collaborative model training with decentralized data. However, the training process of Large Language Models (LLMs) generally incurs the update of significant parameters, which limits the applicability of FL techniques to tackle the LLMs in real scenarios. Prompt tuning can significantly reduce the number of parameters to update, but it either incurs performance degradation or low training efficiency. The straightforward utilization of prompt tuning in the FL often raises non-trivial communication costs and dramatically degrades performance. In addition, the decentralized data is generally non-Independent and Identically Distributed (non-IID), which brings client drift problems and thus poor performance. This paper proposes a Parameter-efficient prompt Tuning approach with Adaptive Optimization, i.e., FedPepTAO, to enable efficient and effective FL of LLMs. First, an efficient partial prompt tuning approach is proposed to improve performance and efficiency simultaneously. Second, a novel adaptive optimization method is developed to address the client drift problems on both the device and server sides to enhance performance further. Extensive experiments based on 10 datasets demonstrate the superb performance (up to 60.8\% in terms of accuracy) and efficiency (up to 97.59\% in terms of training time) of FedPepTAO compared with 9 baseline approaches. Our code is available at https://github.com/llm-eff/FedPepTAO.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- GPT understands, tooAI Open, 2024
- HeterPS: Distributed deep learning with reinforcement learning based scheduling in heterogeneous environmentsFuture Generation Computer Systems, 2023
- Parameter-efficient fine-tuning of large-scale pre-trained language modelsNature Machine Intelligence, 2023
- Large‐scale knowledge distillation with elastic heterogeneous computing resourcesConcurrency and Computation: Practice and Experience, 2022
- Accelerating Federated Learning via Momentum Gradient DescentIEEE Transactions on Parallel and Distributed Systems, 2020
- Efficient Scheduling of Scientific Workflows Using Hot Metadata in a Multisite CloudIEEE Transactions on Knowledge and Data Engineering, 2018
- Lightweight Indexing and Querying Services for Big Spatial DataIEEE Transactions on Services Computing, 2016
- Multi-objective scheduling of Scientific Workflows in multisite cloudsFuture Generation Computer Systems, 2016
- A Survey of Data-Intensive Scientific Workflow ManagementJournal of Grid Computing, 2015
- Annotating Expressions of Opinions and Emotions in LanguageLanguage Resources and Evaluation, 2005