QuestEval: Summarization Asks for Fact-based Evaluation
Authors:
- Thomas ScialomTSThomas Scialom
- Paul-Alexis DrayPDPaul-Alexis Dray
- Jacopo StaianoJSJacopo Staiano
Preprint
Abstract
Summarization evaluation remains an open research problem: current metrics such as ROUGE are known to be limited and to correlate poorly with human judgments. To alleviate this issue, recent work has proposed evaluation metrics which rely on question answering models to assess whether a summary contains all the relevant information in its source document. Though promising, the proposed approaches have so far failed to correlate better than ROUGE with human judgments. In this paper, we extend previous approaches and propose a unified framework, named QuestEval. In contrast to established metrics such as ROUGE or BERTScore, QuestEval does not require any ground-truth reference. Nonetheless, QuestEval substantially improves the correlation with human judgments over four evaluation dimensions (consistency, coherence, fluency, and relevance), as shown in the extensive experiments we report.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Reducing Quantity Hallucinations in Abstractive Summarization2020
- Evaluating the Factual Consistency of Abstractive Text Summarization2020
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization2020
- Asking and Answering Questions to Evaluate the Factual Consistency of Summaries2020
- A Semantic QA-Based Approach for Text Summarization EvaluationProceedings of the AAAI Conference on Artificial Intelligence, 2018
- Why We Need New Evaluation Metrics for NLG2017
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016
- Automatically Assessing Machine Summary Content Without a Gold StandardComputational Linguistics, 2013
- Discourse Constraints for Document CompressionComputational Linguistics, 2010