Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models
Authors:
Preprint
Abstract
Parameter-efficient fine-tuning (PEFT) methods typically assume that Large Language Models (LLMs) are trained on data from a single device or client. However, real-world scenarios often require fine-tuning these models on private data distributed across multiple devices. Federated Learning (FL) offers an appealing solution by preserving user privacy, as sensitive data remains on local devices during training. Nonetheless, integrating PEFT methods into FL introduces two main challenges: communication overhead and data heterogeneity. In this paper, we introduce FedTT and FedTT+, methods for adapting LLMs by integrating tensorized adapters into client-side models' encoder/decoder blocks. FedTT is versatile and can be applied to both cross-silo FL and large-scale cross-device FL. FedTT+, an extension of FedTT tailored for cross-silo FL, enhances robustness against data heterogeneity by adaptively freezing portions of tensor factors, further reducing the number of trainable parameters. Experiments on BERT and LLaMA models demonstrate that our proposed methods successfully address data heterogeneity challenges and perform on par or even better than existing federated PEFT approaches while achieving up to 10$\times$ reduction in communication cost.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Robust Decentralized Learning With Local Updates and Gradient TrackingIEEE Transactions on Networking, 2025
- Parameter-efficient fine-tuning of large-scale pre-trained language modelsNature Machine Intelligence, 2023
- Parameter-Efficient Fine-Tuning without Introducing New Latency2023
- Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System HeterogeneityIEEE Journal on Selected Areas in Information Theory, 2022
- SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapters2022
- BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models2022
- Prefix-Tuning: Optimizing Continuous Prompts for Generation2021
- Parameter-Efficient Transfer Learning with Diff Pruning2021
- Advances and Open Problems in Federated Learning2021
- Tensor-Train DecompositionSIAM Journal on Scientific Computing, 2011