Generative Information Retrieval Evaluation
Authors:
Preprint
Abstract
This paper is a draft of a chapter intended to appear in a forthcoming book on generative information retrieval, co-edited by Chirag Shah and Ryen White. In this chapter, we consider generative information retrieval evaluation from two distinct but interrelated perspectives. First, large language models (LLMs) themselves are rapidly becoming tools for evaluation, with current research indicating that LLMs may be superior to crowdsource workers and other paid assessors on basic relevance judgement tasks. We review past and ongoing related research, including speculation on the future of shared task initiatives, such as TREC, and a discussion on the continuing need for human assessments. Second, we consider the evaluation of emerging LLM-based generative information retrieval (GenIR) systems, including retrieval augmented generation (RAG) systems. We consider approaches that focus both on the end-to-end evaluation of GenIR systems and on the evaluation of a retrieval component as an element in a RAG system. Going forward, we expect the evaluation of GenIR systems to be at least partially based on LLM-based assessment, creating an apparent circularity, with a system seemingly evaluating its own output. We resolve this apparent circularity in two ways: 1) by viewing LLM-based assessment as a form of "slow search", where a slower IR system is used for evaluation and training of a faster production IR system; and 2) by recognizing a continuing need to ground evaluation in human assessment, even if the characteristics of that human assessment must change.Keywords
All Related Versions
This publication has 67 references indexed in Scilit:
- Where a Little Change Makes a Big Difference: A Preliminary Exploration of Children’s Queries2023
- Children's query types and reformulations in Google searchInformation Processing & Management, 2018
- Assessing efficiency–effectiveness tradeoffs in multi-stage retrieval systems without using relevance judgmentsInformation Retrieval Journal, 2016
- Query strategies during information searching: Effects of prior domain knowledge and complexity of the information problems to be solvedInformation Processing & Management, 2015
- Online search in english as a non‐native languageProceedings of the Association for Information Science and Technology, 2015
- A review of factors influencing user satisfaction in information retrievalJournal of the American Society for Information Science and Technology, 2010
- Analysis of multiple query reformulations on the web: The interactive information retrieval contextInformation Processing & Management, 2006
- User-friendly systems instead of user-friendly front-endsJournal of the American Society for Information Science, 1992
- THE PROBABILITY RANKING PRINCIPLE IN IRJournal of Documentation, 1977
- The use of hierarchic clustering in information retrievalInformation Storage and Retrieval, 1971