Does LLM Relevance Labelling Work for Arabic?
Open Access
- 19 July 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 3568-3573
Abstract
Large Language Models (LLMs) are increasingly used in Information Retrieval, both within retrieval pipelines and for constructing evaluation resources. Existing studies on using LLMs for IR evaluation, however, focus almost exclusively on English, leaving their applicability to other languages, where evaluation resources are often limited and highly needed, unexplored. We examine the use of LLMs to generate relevance labels for an Arabic test collection (ArTest). Using about 10K relevance labels, generated by three LLMs, and used to order eight automated systems and nine simulated manual systems, we show that agreement on binary labels and system ordering is generally high, with no cases of significant opposite conclusions; however, there are cases of false or missed improvements and noticeable limitations when labelling manual, highly performing systems. These findings align with results reported for English and indicate that LLMs could, with some caveats, offer a viable approach to supporting IR evaluation in languages with limited evaluation resources.Keywords
This publication has 22 references indexed in Scilit:
- Query–Document Dense Vectors for LLM Relevance Judgment Bias Analysis2026
- Corpus Subsampling: Estimating the Effectiveness of Neural Retrieval Models on Large Corpora2025
- ArTest: The First Test Collection for Arabic Web Search with Relevance Rationales2020
- Gauging the Quality of Relevance Assessments using Inter-Rater Agreement2017
- EveTAR2016
- ArabicWeb162016
- Pooled Evaluation Over Query Variations2015
- Models and metrics2012
- Efficient construction of large test collections1998
- A NEW MEASURE OF RANK CORRELATIONBiometrika, 1938