Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary
Open Access
- 2 August 2021
- journal article
- research article
- Published by MIT Press in Transactions of the Association for Computational Linguistics
- Vol. 9, 774-789
Abstract
A desirable property of a reference-based evaluation metric that measures the content quality of a summary is that it should estimate how much information that summary has in common with a reference. Traditional text overlap based metrics such as ROUGE fail to achieve this because they are limited to matching tokens, either lexically or via embeddings. In this work, we propose a metric to evaluate the content quality of a summary using question-answering (QA). QA-based methods directly measure a summary’s information overlap with a reference, making them fundamentally different than text overlap metrics. We demonstrate the experimental benefits of QA-based metrics through an analysis of our proposed metric, QAEval. QAEval outperforms current state-of-the-art metrics on most evaluations using benchmark datasets, while being competitive on others due to limitations of state-of-the-art models. Through a careful analysis of each component of QAEval, we identify its performance bottlenecks and estimate that its potential upper-bound performance surpasses all other automatic metrics, approaching that of the gold-standard Pyramid Method. 1Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization2020
- Asking and Answering Questions to Evaluate the Factual Consistency of Summaries2020
- MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance2019
- Know What You Don’t Know: Unanswerable Questions for SQuAD2018
- Ranking Sentences for Extractive Summarization with Reinforcement Learning2018
- Learning to Score System Summaries for Better Content Selection Evaluation.2017
- Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond2016
- Meteor Universal: Language Specific Translation Evaluation for Any Target Language2014
- Automatically Assessing Machine Summary Content Without a Gold StandardComputational Linguistics, 2013
- Summarization system evaluation revisitedACM Transactions on Speech and Language Processing, 2008