Towards Practical Few-shot Federated NLP
Authors:
- Dongqi CaiDCDongqi Cai
- Yaozong WuYWYaozong Wu
- Haitao YuanHYHaitao Yuan
- Shangguang WangSWShangguang Wang
- Felix Xiaozhu LinFLFelix Xiaozhu Lin
- Mengwei XuMXMengwei Xu
Preprint
Abstract
Transformer-based pre-trained models have emerged as the predominant solution for natural language processing (NLP). Fine-tuning such pre-trained models for downstream tasks often requires a considerable amount of labeled private data. In practice, private data is often distributed across heterogeneous mobile devices and may be prohibited from being uploaded. Moreover, well-curated labeled data is often scarce, presenting an additional challenge. To address these challenges, we first introduce a data generator for federated few-shot learning tasks, which encompasses the quantity and skewness of scarce labeled data in a realistic setting. Subsequently, we propose AUG-FedPrompt, a prompt-based federated learning system that exploits abundant unlabeled data for data augmentation. Our experiments indicate that AUG-FedPrompt can perform on par with full-set fine-tuning with a limited amount of labeled data. However, such competitive performance comes at a significant system cost.Keywords
All Related Versions
- Version 1, 2022-12-01, ArXiv
- Version 2, 2023-08-19, ArXiv
- Published version: , 42.
This publication has 8 references indexed in Scilit:
- Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System HeterogeneityIEEE Journal on Selected Areas in Information Theory, 2022
- True Few-Shot Learning with Prompts—A Real-World PerspectiveTransactions of the Association for Computational Linguistics, 2022
- Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference2021
- Client Selection and Bandwidth Allocation in Wireless Federated Learning Networks: A Long-Term PerspectiveIEEE Transactions on Wireless Communications, 2020
- Transformer-Based Neural Network for Answer Selection in Question AnsweringIEEE Access, 2019
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Deep learning for sentiment analysis: A surveyWIREs Data Mining and Knowledge Discovery, 2018
- A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference2018