A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods
Open Access
- 27 October 2021
- journal article
- research article
- Published by MIT Press in Transactions of the Association for Computational Linguistics
- Vol. 9, 1132-1146
Abstract
The quality of a summarization evaluation metric is quantified by calculating the correlation between its scores and human annotations across a large number of summaries. Currently, it is unclear how precise these correlation estimates are, nor whether differences between two metrics’ correlations reflect a true difference or if it is due to mere chance. In this work, we address these two problems by proposing methods for calculating confidence intervals and running hypothesis tests for correlations using two resampling methods, bootstrapping and permutation. After evaluating which of the proposed methods is most appropriate for summarization through two simulation experiments, we analyze the results of applying these methods to several different automatic evaluation metrics across three sets of human annotations. We find that the confidence intervals are rather wide, demonstrating high uncertainty in the reliability of automatic metrics. Further, although many metrics fail to show statistical improvements over ROUGE, two recent works, QAEval and BERTScore, do so in some evaluation settings. 1Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- Meteor Universal: Language Specific Translation Evaluation for Any Target Language2014
- Testing for Significance of Increased Correlation with Human Judgment2014
- Summary Evaluation: Together We Stand NPowER-ed2013
- Better Metrics to Automatically Predict the Quality of a Text SummaryAlgorithms, 2012
- Summarization system evaluation revisitedACM Transactions on Speech and Language Processing, 2008
- Sample Size Requirements for Estimating Pearson, Kendall and Spearman CorrelationsPsychometrika, 2000
- Statistical Methods for Research Workers1992
- Comparison of Tests of the Equality of Dependent Correlation CoefficientsJournal of the American Statistical Association, 1971
- An Analysis of Variance Test for Normality (Complete Samples)Biometrika, 1965
- An analysis of variance test for normality (complete samples)Biometrika, 1965