Towards Practical Few-shot Federated NLP
Open Access
- 8 May 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 42-48
Abstract
Transformer-based pre-trained models have emerged as the predominant solution for natural language processing (NLP). Fine-tuning such pre-trained models for downstream tasks often requires a considerable amount of labeled private data. In practice, private data is often distributed across heterogeneous mobile devices and may be prohibited from being uploaded. Moreover, well-curated labeled data is often scarce, presenting an additional challenge. To address these challenges, we first introduce a data generator for federated few-shot learning tasks, which encompasses the quantity and skewness of scarce labeled data in a realistic setting. Subsequently, we propose AUG-FedPrompt, a prompt-based federated learning system that exploits abundant unlabeled data for data augmentation. Our experiments indicate that AUG-FedPrompt can perform on par with full-set fine-tuning with a limited amount of labeled data. However, such competitive performance comes at a significant system cost.Keywords
All Related Versions
Funding Information
- National Key Research and Development Program of China (2020YFB1805500)
- the Fundamental Research Funds for the Central Universities
- NSFC (62032003, 61922017, 61921003, 62102045)
- Beijing Nova Program (Z211100002121118)
- Young Elite Scientists Sponsorship Program by CAST (2021QNRC001)
This publication has 13 references indexed in Scilit:
- Towards Mitigating Device Heterogeneity in Federated Learning via Adaptive Model Quantization2021
- How many data points is a prompt worth?2021
- IntelliCode compose: code generation using transformer2020
- Capuchin2020
- Federated LearningSynthesis Lectures on Artificial Intelligence and Machine Learning, 2019
- How Does BERT Answer Questions?2019
- Transformer-Based Neural Network for Answer Selection in Question AnsweringIEEE Access, 2019
- Deep learning for sentiment analysis: A surveyWIREs Data Mining and Knowledge Discovery, 2018
- A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference2018
- Curriculum learning2009