GenAug: Data Augmentation for Finetuning Text Generators
Preprint
Abstract
In this paper, we investigate data augmentation for text generation, which we call GenAug. Text generation and language modeling are important tasks within natural language processing, and are especially challenging for low-data regimes. We propose and evaluate various augmentation methods, including some that incorporate external knowledge, for finetuning GPT-2 on a subset of Yelp Reviews. We also examine the relationship between the amount of augmentation and the quality of the generated text. We utilize several metrics that evaluate important aspects of the generated text including its diversity and fluency. Our experiments demonstrate that insertion of character-level synthetic noise and keyword replacement with hypernyms are effective augmentation methods, and that the quality of generations improves to a peak at approximately three times the amount of original data.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- Evaluating the Evaluation of Diversity in Natural Language Generation2021
- ALOHA: Artificial Learning of Human Attributes for Dialogue AgentsProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Data Augmentation using Pre-trained Transformer Models2020
- Good-Enough Compositional Data Augmentation2020
- On Faithfulness and Factuality in Abstractive Summarization2020
- (Male, Bachelor) and (Female, Ph.D) have different connotations: Parallelly Annotated Stylistic Language Dataset with Multiple Personas2019
- Keep Calm and Switch On! Preserving Sentiment and Fluency in Semantic Text Exchange2019
- Texygen2018
- Adversarial Examples for Evaluating Reading Comprehension Systems2017
- Enhancing Text Categorization with Semantic-enriched Representation and Training Data AugmentationJournal of the American Medical Informatics Association, 2006