Generate, Annotate, and Learn: NLP with Synthetic Text
Open Access
- 12 August 2022
- journal article
- research article
- Published by MIT Press in Transactions of the Association for Computational Linguistics
- Vol. 10, 826-842
Abstract
This paper studies the use of language models as a source of synthetic unlabeled text for NLP. We formulate a general framework called “generate, annotate, and learn (GAL)” to take advantage of synthetic text within knowledge distillation, self-training, and few-shot learning applications. To generate high-quality task-specific text, we either fine-tune LMs on inputs from the task of interest, or prompt large LMs with few examples. We use the best available classifier to annotate synthetic text with soft pseudo labels for knowledge distillation and self-training, and use LMs to obtain hard labels for few-shot learning. We train new supervised models on the combination of labeled and pseudo-labeled data, which results in significant gains across several applications. We investigate key components of GAL and present theoretical and empirical arguments against the use of class-conditional LMs to generate synthetic labeled text instead of unlabeled text. GAL achieves new state-of-the-art knowledge distillation results for 6-layer transformers on the GLUE leaderboard.Keywords
All Related Versions
This publication has 38 references indexed in Scilit:
- Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond2016
- Improving Neural Machine Translation Models with Monolingual Data2016
- That's So Annoying!!!: A Lexical and Frame-Semantic Embedding Based Data Augmentation Approach to Automatic Categorization of Annoying Behaviors using #petpeeve Tweets2015
- A Neural Attention Model for Abstractive Sentence Summarization2015
- Building a Semantic Parser Overnight2015
- Model compression2006
- Bootstrapping without the boot2005
- Understanding the Yarowsky AlgorithmComputational Linguistics, 2004
- Unsupervised word sense disambiguation rivaling supervised methods1995
- Probability of error of some adaptive pattern-recognition machinesIEEE Transactions on Information Theory, 1965