A Field Guide to Automatic Evaluation of LLM-Generated Summaries
Open Access
- 10 July 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 2832-2836
Abstract
Large Language models (LLMs) are rapidly being adopted for tasks such as text summarization, in a wide range of industries. This has driven the need for scalable, automatic, reliable, and cost-effective methods to evaluate the quality of LLM-generated text. What is meant by evaluating an LLM is not yet well defined and there are widely different expectations about what kind of information evaluation will produce. Evaluation methods that were developed for traditional Natural Language Processing (NLP) tasks (before the rise of LLMs) remain applicable but are not sufficient for capturing high-level semantic qualities of summaries. Emerging evaluation methods that use LLMs to evaluate LLM-output, appear to be powerful but lacking in reliability. New elements of LLM generated text that were not an element of previous NLP tasks, such as the artifacts of hallucination, need to be considered. We outline the different types of LLM evaluation currently used in the literature but focus on offline, system-level evaluation of the text generated by LLMs. Evaluating LLM-generated summaries is a complex and fast-evolving area, and we propose strategies for applying evaluation methods to avoid common pitfalls. Despite having promising strategies for evaluating LLM summaries, we highlight some open challenges that remain.Keywords
This publication has 19 references indexed in Scilit:
- On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top- n Recommendation2024
- How Well do Offline Metrics Predict Online Performance of Product Ranking Models?2023
- SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in SummarizationTransactions of the Association for Computational Linguistics, 2022
- On the Limitations of Reference-Free Evaluations of Generated Text2022
- Domain-Specific Language Model Pretraining for Biomedical Natural Language ProcessingACM Transactions on Computing for Healthcare, 2021
- QuestEval: Summarization Asks for Fact-based Evaluation2021
- Evaluating the Factual Consistency of Abstractive Text Summarization2020
- Re-evaluating Evaluation in Text Summarization2020
- Answers Unite! Unsupervised Metrics for Reinforced Summarization Models2019
- ROUGE-C: A fully automated evaluation method for multi-document summarization2008