Robust and Private Federated Learning on LLMs
Open Access
- 12 April 2024
- book chapter
- Published by Springer Nature
- p. 189-196
Abstract
Large Language Models (LLMs) have gained significant attention in recent years due to their potential to revolutionize various industries and sectors. However, scaling LLMs further requires access to substantial linguistic resources that are being rapidly depleted. Moreover, the available text sources such as emails, social media interactions, or internal documents may contain private information, making them susceptible to misuse. On-premises Federated Learning (FL) with privacy-preserving model updates is an alternative avenue for LLMs’ development that ensures data sovereignty and enables peers to collaborate while ensuring that the sensitive parts of their private data cannot be reconstructed. However, in the case of large-scale FL, there is also a risk of malicious users attempting to poison LLMs for their benefit. The problem of protecting the learning procedure against such users is known as Byzantine-robustness, and it is crucial to develop models that perform accurately despite faulty machines and poisonous data. Designing FL methods that are simultaneously privacy-preserving and Byzantine-robust is challenging. However, ongoing research suggests ways to incorporate the differentially-private Gaussian mechanism for privacy preservation and spectral robust-averaging for robustness. However, whether this approach applies to LLMs or whether a major player in the domain would emerge and capture all private information sources through network effects remains to be seen.Keywords
Funding Information
- ArmaSuisse
This publication has 19 references indexed in Scilit:
- A review of applications in federated learningComputers & Industrial Engineering, 2020
- Robust Estimation via Robust Gradient EstimationJournal of the Royal Statistical Society Series B: Statistical Methodology, 2020
- Exploiting Unintended Feature Leakage in Collaborative Learning2019
- Deep Models Under the GAN2017
- Rényi Differential Privacy2017
- Privacy-Preserving Deep Learning: Revisited and Enhanced2017
- Learning from untrusted data2017
- Is Interaction Necessary for Distributed Private Learning?2017
- Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures2015
- Multivariate Estimation with High Breakdown Point1985