FedBERT: When Federated Learning Meets Pre-training

Authors:
Top Cited Papers
Open Access
Publisher website
Google Scholar
Abstract
The fast growth ofpre-trained models(PTMs) has brought natural language processing to a new era, which has become a dominant technique for variousnatural language processing(NLP) applications. Every user can download the weights of PTMs, then fine-tune the weights for a task on the local side. However, the pre-training of a model relies heavily on accessing a large-scale of training data and requires a vast amount of computing resources. These strict requirements make it impossible for any single client to pre-train such a model. To grant clients with limited computing capability to participate in pre-training a large model, we propose a new learning approach,FedBERT, that takes advantage of the federated learning and split learning approaches, resorting to pre-training BERT in a federated way.FedBERTcan prevent sharing the raw data information and obtain excellent performance. Extensive experiments on seven GLUE tasks demonstrate thatFedBERTcan maintain its effectiveness without communicating to the sensitive local data of clients.
Funding Information
  • National Natural Science Foundation of China (62072204, and 62102157)
  • Fundamental Research Funds for the Central Universities (HUST:2020kfyXJJS019)

This publication has 20 references indexed in Scilit: