Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models
Preprint
Abstract
Recent models for unsupervised representation learning of text have employed a number of techniques to improve contextual word representations but have put little focus on discourse-level representations. We propose CONPONO, an inter-sentence objective for pretraining language models that models discourse coherence and the distance between sentences. Given an anchor sentence, our model is trained to predict the text k sentences away using a sampled-softmax objective where the candidates consist of neighboring sentences and sentences randomly sampled from the corpus. On the discourse representation benchmark DiscoEval, our model improves over the previous state-of-the-art by up to 13% and on average 4% absolute across 7 tasks. Our model is the same size as BERT-Base, but outperforms the much larger BERT- Large model and other more recent approaches that incorporate discourse. We also show that CONPONO yields gains of 2%-6% absolute even for tasks that do not explicitly evaluate discourse: textual entailment (RTE), common sense reasoning (COPA) and reading comprehension (ReCoRD).Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations2019
- A Cross-Domain Transferable Neural Coherence Model2019
- DisSent: Learning Sentence Representations from Explicit Discourse Relations2019
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding2018
- Deep Contextualized Word Representations2018
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016
- The third PASCAL recognizing textual entailment challenge2007