DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence
Authors:
- Wei ZhaoWZWei Zhao
- Steffen EgerSESteffen Eger
Preprint
Abstract
Recently, there has been a growing interest in designing text generation systems from a discourse coherence perspective, e.g., modeling the interdependence between sentences. Still, recent BERT-based evaluation metrics are weak in recognizing coherence, and thus are not reliable in a way to spot the discourse-level improvements of those text generation systems. In this work, we introduce DiscoScore, a parametrized discourse metric, which uses BERT to model discourse coherence from different perspectives, driven by Centering theory. Our experiments encompass 16 non-discourse and discourse metrics, including DiscoScore and popular coherence models, evaluated on summarization and document-level machine translation (MT). We find that (i) the majority of BERT-based metrics correlate much worse with human rated coherence than early discourse metrics, invented a decade ago; (ii) the recent state-of-the-art BARTScore is weak when operated at system level -- which is particularly problematic as systems are typically compared in this manner. DiscoScore, in contrast, achieves strong system-level correlation with human ratings, not only in coherence but also in factual consistency and other aspects, and surpasses BARTScore by over 10 correlation points on average. Further, aiming to understand DiscoScore, we provide justifications to the importance of discourse coherence for evaluation metrics, and explain the superiority of one variant over another. Our code is available at \url{https://github.com/AIPHES/DiscoScore}.Keywords
All Related Versions
This publication has 39 references indexed in Scilit:
- Multilingual Denoising Pre-training for Neural Machine TranslationTransactions of the Association for Computational Linguistics, 2020
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation2020
- FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization2020
- Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models2020
- BLEURT: Learning Robust Metrics for Text Generation2020
- SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization2020
- Discourse-Aware Neural Extractive Text Summarization2020
- Using Context in Neural Machine Translation Training Objectives2020
- Modeling Local Coherence: An Entity-Based ApproachComputational Linguistics, 2008