DocAsRef: An Empirical Study on Repurposing Reference-Based Summary Quality Metrics Reference-Freely
Authors:
Preprint
Abstract
Automated summary quality assessment falls into two categories: reference-based and reference-free. Reference-based metrics, historically deemed more accurate due to the additional information provided by human-written references, are limited by their reliance on human input. In this paper, we hypothesize that the comparison methodologies used by some reference-based metrics to evaluate a system summary against its corresponding reference can be effectively adapted to assess it against its source document, thereby transforming these metrics into reference-free ones. Experimental results support this hypothesis. After being repurposed reference-freely, the zero-shot BERTScore using the pretrained DeBERTa-large-MNLI model of <0.5B parameters consistently outperforms its original reference-based version across various aspects on the SummEval and Newsroom datasets. It also excels in comparison to most existing reference-free metrics and closely competes with zero-shot summary evaluators based on GPT-3.5.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment2023
- SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in SummarizationTransactions of the Association for Computational Linguistics, 2022
- SueNes: A Weakly Supervised Approach to Evaluating Single-Document Summarization via Negative Sampling2022
- Reference-free Summarization Evaluation via Semantic Correlation and Compression Ratio2022
- SummEval: Re-evaluating Summarization EvaluationTransactions of the Association for Computational Linguistics, 2021
- Fill in the BLANC: Human-free quality estimation of document summaries2020
- BLEURT: Learning Robust Metrics for Text Generation2020
- SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization2020
- Answers Unite! Unsupervised Metrics for Reinforced Summarization Models2019
- MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance2019