SummEval: Re-evaluating Summarization Evaluation
Preprint
Abstract
The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continues to inhibit progress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 12 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd-sourced human annotations, 2) we consistently benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics, 3) we assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset and share it in a unified format, 4) we implement and share a toolkit that provides an extensible and unified API for evaluating summarization models across a broad range of automatic metrics, 5) we assemble and share the largest and most diverse, in terms of model types, collection of human judgments of model-generated summaries on the CNN/Daily Mail dataset annotated by both expert judges and crowd source workers. We hope that this work will help promote a more complete evaluation protocol for text summarization as well as advance research in developing evaluation metrics that better correlate with human judgements.Keywords
All Related Versions
This publication has 38 references indexed in Scilit:
- SacreROUGE: An Open-Source Library for Using and Developing Summarization Evaluation Metrics2020
- Evaluating the Factual Consistency of Abstractive Text Summarization2020
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- Content Selection in Deep Learning Models of Summarization2018
- Neural Latent Extractive Document Summarization2018
- A Graph-theoretic Summary Evaluation for ROUGE2018
- Improving Abstraction in Text Summarization2018
- On the Abstractiveness of Neural Document Summarization2018
- Better Summarization Evaluation with Word Embeddings for ROUGE2015
- Re-evaluating Automatic Summarization with BLEU and 192 Shades of ROUGE2015