FedHLT: Efficient Federated Low-Rank Adaption with Hierarchical Language Tree for Multilingual Modeling
Open Access
- 13 May 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1558-1567
Abstract
Federated Multilingual Modeling (FMM) has become an essential approach in natural language processing (NLP) due to increasing linguistic diversity and the heightened emphasis on data privacy. However, FMM faces two primary challenges: 1) the high communication costs inherent in network operations, and 2) the complexities arising from parameter interference, as languages exhibit both unique characteristics and shared features. To tackle these issues, we introduce a communication-efficient framework for Multilingual Modeling (MM) that combines low-rank adaptation with a hierarchical language tree structure. Our method maintains the base model's weights while focusing on updating only the Low-rank adaptation (LoRA) parameters, significantly reducing communication costs. Additionally, we mitigate parameter conflicts by organizing languages based on their familial ties rather than merging all LoRA parameters together. Our experimental findings reveal that this novel model surpasses established baseline models in performance and markedly decreases communication overhead.Keywords
Funding Information
- RGC GRF (2151185)
- CUHK (14222922)
This publication has 26 references indexed in Scilit:
- Efficient Distribution Similarity Identification in Clustered Federated Learning via Principal Angles between Client Data SubspacesProceedings of the AAAI Conference on Artificial Intelligence, 2023
- A Survey of Trustworthy Federated Learning with Perspectives on Security, Robustness and Privacy2023
- Federated Learning of Gboard Language Models with Differential Privacy2023
- FedSoft: Soft Clustered Federated Learning with Proximal Local UpdatingProceedings of the AAAI Conference on Artificial Intelligence, 2022
- Modular and Parameter-Efficient Fine-Tuning for NLP Models2022
- Pretrained Models for Multilingual Federated Learning2022
- BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models2022
- S4-Tuning: A Simple Cross-lingual Sub-network Tuning Method-Tuning: A Simple Cross-lingual Sub-network Tuning Method2022
- Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning2021
- Prefix-Tuning: Optimizing Continuous Prompts for Generation2021