Federated Large Language Model: Solutions, Challenges and Future Directions
Authors:
- 23 October 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Wireless Communications
- Vol. 32 (4) , 82-89
Abstract
Large language models (LLMs) have become increasingly popular due to their exceptional performance in various artificial intelligence applications. However, their development often suffers from the scarcity of high-quality data and the extensive requirements for computing resources. These obstacles are even more severe for enterprises in vertical industries, which have limited computer resources but urgently require large-scale models for specific activities. To address these issues, LLMs call for the integration of federated learning (FL), which enables the collaborative learning of a powerful LLM using private data and computing resources from multiple entities. In this article, we present a systematic introduction to the federated large language model (Fed-LLM), a distributed learning of LLM in the FL manner. We first introduce the learning paradigm of Fed-LLM, which is called federated parameter-efficient fine-tuning (Fed-PEFT). Fed-PEFT empowers the collaborative fine-tuning of pre-trained LLMs by only involving a small subset of parameters in local LLMs. Specifically, we detail the workflow of Fed-PEFT, and summarize the state-of-the-art solutions in this area. Additionally, we discuss the challenges faced in Fed-LLMs, including efficiency, privacy, and security. Finally, we introduce future directions to facilitate the research of Fed-LLMs and guide coming explorations in this nascent field.Keywords
Funding Information
- National Natural Science Foundation of China (62122066,U20A20182,62302164)
- China Postdoctoral Science Foundation (2023M731071)
This publication has 8 references indexed in Scilit:
- Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generation2023
- pFedPrompt: Learning Personalized Prompt for Vision-Language Models in Federated Learning2023
- Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization2023
- FedPETuning: When Federated Learning Meets the Parameter-Efficient Tuning Methods of Pre-trained Language Models2023
- Client-Customized Adaptation for Parameter-Efficient Federated Learning2023
- Parameter-Efficient Fine-Tuning without Introducing New Latency2023
- BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models2022
- Prefix-Tuning: Optimizing Continuous Prompts for Generation2021