Generate, Annotate, and Learn: NLP with Synthetic Text
Preprint
Abstract
Semi-Supervised Learning (SSL) has seen success in many application domains, but this success often hinges on the availability of task-specific unlabeled data. Knowledge distillation (KD) has enabled effective optimization of compact neural nets, achieving the best results when the knowledge of an expensive network is distilled via fresh task-specific unlabeled data. However, task-specific unlabeled data can be challenging to find, especially for NLP. We investigate the use of generative models in synthesizing unlabeled data and present a simple and general framework called "generate, annotate, and learn (GAL)". A language model (LM) is used to synthesize in-domain unlabeled data. Then, a classifier is used to annotate such data. Finally, synthetically generated and annotated data is used to advance SSL, KD, and few-shot learning on NLP and tabular tasks. To obtain a strong task-specific LM, we either fine-tune a large LM on inputs from a specific task, or prompt a large LM with a few input examples and conditionally generate more unlabeled examples. It also yields a new state-of-the-art for 6-layer transformers on the GLUE leaderboard. Finally, self-training with GAL offers large gains on four tabular tasks from the UCI repository.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation2021
- Making Pre-trained Language Models Better Few-shot Learners2021
- Multiclass classification of dry beans using computer vision and machine learning techniquesComputers and Electronics in Agriculture, 2020
- Data Augmentation using Pre-trained Transformer Models2020
- TinyBERT: Distilling BERT for Natural Language Understanding2020
- Transformers: State-of-the-Art Natural Language Processing2020
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- Corpora Generation for Grammatical Error Correction2019
- Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations2018
- Learning with a probabilistic teacherIEEE Transactions on Information Theory, 1970