Do LLM-judges Align with Human Relevance in Cranfield-style Recommender Evaluation?
Authors:
- Gustavo PenhaGPGustavo Penha
- Aleksandr V. PetrovAPAleksandr V. Petrov
- Claudia HauffCHClaudia Hauff
- Enrico PalumboEPEnrico Palumbo
- Ali VardasbiAVAli Vardasbi
- Edoardo D'AmicoEDEdoardo D'Amico
- Francesco FabbriFFFrancesco Fabbri
- Alice WangAWAlice Wang
- Praveen ChandarPCPraveen Chandar
- Henrik LindstromHLHenrik Lindstrom
- Hugues BouchardHBHugues Bouchard
- Mounia LalmasMLMounia Lalmas
Preprint
Abstract
Evaluating recommender systems remains a long-standing challenge, as offline methods based on historical user interactions and train-test splits often yield unstable and inconsistent results due to exposure bias, popularity bias, sampled evaluations, and missing-not-at-random patterns. In contrast, textual document retrieval benefits from robust, standardized evaluation via Cranfield-style test collections, which combine pooled relevance judgments with controlled setups. While recent work shows that adapting this methodology to recommender systems is feasible, constructing such collections remains costly due to the need for manual relevance judgments, thus limiting scalability. This paper investigates whether Large Language Models (LLMs) can serve as reliable automatic judges to address these scalability challenges. Using the ML-32M-ext Cranfield-style movie recommendation collection, we first examine the limitations of existing evaluation methodologies. Then we explore the alignment and the recommender systems ranking agreement between the LLM-judge and human provided relevance labels. We find that incorporating richer item metadata and longer user histories improves alignment, and that LLM-judge yields high agreement with human-based rankings (Kendall's tau = 0.87). Finally, an industrial case study in the podcast recommendation domain demonstrates the practical value of LLM-judge for model selection. Overall, our results show that LLM-judge is a viable and scalable approach for evaluating recommender systems.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- A systematic literature review of recent advances on context-aware recommender systemsArtificial Intelligence Review, 2024
- Large Language Models are Zero-Shot Rankers for Recommender Systems2024
- Shallow pooling for sparse labelsInformation Retrieval Journal, 2022
- Streamlining Evaluation with ir-measures2022
- Offline evaluation options for recommender systemsInformation Retrieval Journal, 2020
- Statistical biases in Information Retrieval metrics for recommender systemsInformation Retrieval Journal, 2017
- Contrasting Offline and Online Results when Evaluating Recommendation Algorithms2016
- The MovieLens DatasetsACM Transactions on Interactive Intelligent Systems, 2015
- Dimensions of relevanceInformation Processing & Management, 2000
- How many relevances in information retrieval?Interacting with Computers, 1998