A Review of Faithfulness Metrics for Hallucination Assessment in Large Language Models
Top Cited Papers
Open Access
- 12 June 2025
- journal article
- review article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Signal Processing
- Vol. 19 (7) , 1362-1375
Abstract
This review examines the means with which faithfulness has been evaluated across open-ended summarization, question answering and machine translation tasks. We find that the use of Large Language Models (LLMs) as a faithfulness evaluator is commonly the metric that is most highly correlated with human judgement. The means with which other studies have mitigated hallucinations is discussed, with both retrieval augmented generation (RAG) and prompting framework approaches having been linked with superior faithfulness, whilst other recommendations for mitigation are provided. Research into faithfulness is integral to the continued widespread use of LLMs, as unfaithful responses can pose major risks to many areas whereby LLMs would otherwise be suitable. Furthermore, evaluating open-ended generation provides a more comprehensive measure of LLM performance than commonly used multiplechoice benchmarking, which can help in advancing the trust that can be placed within LLMs.Keywords
All Related Versions
Funding Information
- European Commission
- UK Research and Innovation
- Innovate UK
This publication has 33 references indexed in Scilit:
- Generative artificial intelligence: a systematic review and applicationsMultimedia Tools and Applications, 2024
- Evaluating LLMs on document-based QA: Exact answer selection and numerical extraction using CogTale datasetNatural Language Processing Journal, 2024
- Evaluating Correctness and Faithfulness of Instruction-Following Models for Question AnsweringTransactions of the Association for Computational Linguistics, 2024
- Survey of Hallucination in Natural Language GenerationACM Computing Surveys, 2023
- SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in SummarizationTransactions of the Association for Computational Linguistics, 2022
- Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic LanguagesTransactions of the Association for Computational Linguistics, 2022
- SummEval: Re-evaluating Summarization EvaluationTransactions of the Association for Computational Linguistics, 2021
- On Faithfulness and Factuality in Abstractive Summarization2020
- Natural Questions: A Benchmark for Question Answering ResearchTransactions of the Association for Computational Linguistics, 2019
- Assessing The Factual Accuracy of Generated Text2019