Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
Authors:
Top Cited Papers
Open Access
- 1 December 2020
- journal article
- research article
- Published by MIT Press in Transactions of the Association for Computational Linguistics
- Vol. 8, 264-280
Abstract
Unsupervised pre-training of large neural models has recently revolutionized Natural Language Processing. By warm-starting from the publicly released checkpoints, NLP practitioners have pushed the state-of-the-art on multiple benchmarks while saving significant amounts of compute time. So far the focus has been mainly on the Natural Language Understanding tasks. In this paper, we demonstrate the efficacy of pre-trained checkpoints for Sequence Generation. We developed a Transformer-based sequence-to-sequence model that is compatible with publicly available pre-trained BERT, GPT-2, and RoBERTa checkpoints and conducted an extensive empirical study on the utility of initializing our model, both encoder and decoder, with these checkpoints. Our models result in new state-of-the-art results on Machine Translation, Text Summarization, Sentence Splitting, and Sentence Fusion.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- What Does BERT Learn about the Structure of Language?2019
- Understanding Back-Translation at Scale2018
- SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference2018
- SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing2018
- Bottom-Up Abstractive Summarization2018
- Learning To Split and Rephrase From Wikipedia Edit History2018
- Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization2018
- Optimizing Statistical Machine Translation for Text SimplificationTransactions of the Association for Computational Linguistics, 2016
- Best-Worst Scaling2015
- Long Short-Term MemoryNeural Computation, 1997