Reliable Information Retrieval Systems Performance Evaluation: A Review
Open Access
- 13 March 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 12, 51740-51751
Abstract
With the progressive and availability of various search tools, interest in the evaluation of information retrieval based on user perspective has grown tremendously among researchers. The Information Retrieval System Evaluation is done through Cranfield-paradigm in which the test collections provide the foundation of the evaluation process. The test collections consist of a document corpus, topics, and a set of relevant judgments. The relevant judgments are the documents which retrieved from the test collections based on the topics. The accuracy of the evaluation process is based on the number of relevant documents in the relevance judgment set, called qrels. This paper presents a comprehensive study, which discusses the various ways to improve the number of relevant documents in the qrels to improve the quality of qrels and through that increase the accuracy of the evaluation process. Different ways in which each methodology was performed to retrieve more relevant documents were categorized, described, and analyzed, resulting in an inclusive flow of these methodologies.Keywords
Funding Information
- Ministry of Higher Education Malaysia via Fundamental Research Grant Scheme (FRGS/1.2020/ICT06/UM/02/1)
- UM International Collaboration (ST080-2022)
This publication has 62 references indexed in Scilit:
- Shallow pooling for sparse labelsInformation Retrieval Journal, 2022
- Fewer topics? A million topics? Both?! On topics subsets in test collectionsInformation Retrieval Journal, 2019
- A rank fusion approach based on score distributions for prioritizing relevance assessments in information retrieval evaluationInformation Fusion, 2018
- Do Easy Topics Predict Effectiveness Better Than Difficult Topics?2017
- Information retrieval evaluation using test collectionsInformation Retrieval Journal, 2016
- Pooling-based continuous evaluation of information retrieval systemsInformation Retrieval Journal, 2015
- Rank-Biased Precision Reloaded: Reproducibility and Generalization2015
- The Good, the Bad, the Difficult, and the Easy: Something Wrong with Information Retrieval Evaluation?2008
- Bias and the limits of pooling for large collectionsInformation Retrieval Journal, 2007
- The Philosophy of Information Retrieval Evaluation2002