PreSumm: Predicting Summarization Performance Without Summarizing
Preprint
Abstract
Despite recent advancements in automatic summarization, state-of-the-art models do not summarize all documents equally well, raising the question: why? While prior research has extensively analyzed summarization models, little attention has been given to the role of document characteristics in influencing summarization performance. In this work, we explore two key research questions. First, do documents exhibit consistent summarization quality across multiple systems? If so, can we predict a document's summarization performance without generating a summary? We answer both questions affirmatively and introduce PreSumm, a novel task in which a system predicts summarization performance based solely on the source document. Our analysis sheds light on common properties of documents with low PreSumm scores, revealing that they often suffer from coherence issues, complex content, or a lack of a clear main theme. In addition, we demonstrate PreSumm's practical utility in two key applications: improving hybrid summarization workflows by identifying documents that require manual summarization and enhancing dataset quality by filtering outliers and noisy documents. Overall, our findings highlight the critical role of document properties in summarization performance and offer insights into the limitations of current systems that could serve as the basis for future improvements.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Benchmarking Large Language Models for News SummarizationTransactions of the Association for Computational Linguistics, 2024
- PreQuEL: Quality Estimation of Machine Translation Outputs in Advance2022
- Exploring Neural Models for Query-Focused Summarization2022
- Read Top News First: A Document Reordering Approach for Multi-Document News Summarization2022
- SummEval: Re-evaluating Summarization EvaluationTransactions of the Association for Computational Linguistics, 2021
- Formulating Neural Sentence Ordering as the Asymmetric Traveling Salesman Problem2021
- Fill in the BLANC: Human-free quality estimation of document summaries2020
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- GA, MR, FFNN, PNN and GMM based models for automatic text summarizationComputer Speech & Language, 2009
- A new readability yardstick.Journal of Applied Psychology, 1948