On the Use of LLMs for Relevance Labelling
Open Access
- 17 April 2026
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Information Systems
- Vol. 44 (4) , 1-31
Abstract
Large Language Models (LLMs) are increasingly used to replace human judges to assess the relevance of information objects, raising concerns about circularity, bias, and whether simulated preferences can substitute for human judgement. This work presents experiments using multiple LLMs to label passages for relevance. It examines their gullibility—how easily they are misled into labelling irrelevant passages as relevant. It also compares LLMs with human judges in ranking systems, analysing differences in discriminative power and whether some systems benefit under LLM-based evaluation. Results show that LLMs are influenced by the presence of query terms, even with irrelevant or random passages. Moreover, LLM-generated rankings are highly correlated with those of human judges, with strong agreement on which system is better in pairwise comparisons. However, LLMs may exhibit lower discriminative power, as seen in flatter ranking slopes and missed significance for meaningful improvements. Yet, there are no cases where capable LLMs and human judges reach opposing conclusions with significance. LLMs may boost traditional systems more than neural ones, adding a new concern of system bias. These findings highlight the strong potential of LLMs for relevance labelling, while also highlighting failure cases that call for careful adoption and further research to maintain evaluation integrity. 1Keywords
Funding Information
- Australian Research Council (DP190101113)
This publication has 30 references indexed in Scilit:
- Crowd Worker Strategies in Relevance Judgment Tasks2020
- Understanding Reading Attention Distribution during Relevance Judgement2018
- Gauging the Quality of Relevance Assessments using Inter-Rater Agreement2017
- Models and metrics2012
- Quantifying test collection quality based on the consistency of relevance judgements2011
- How evaluator domain expertise affects search result relevance judgments2008
- Relevance assessment2008
- Variations in relevance judgments and the measurement of retrieval effectiveness1998
- Efficient construction of large test collections1998
- A NEW MEASURE OF RANK CORRELATIONBiometrika, 1938