Preprint
Abstract
Using large language models (LLMs) to predict relevance judgments has shown promising results. Most studies treat this task as a distinct research line, e.g., focusing on prompt design for predicting relevance labels given a query and passage. However, predicting relevance judgments is essentially a form of relevance prediction, a problem extensively studied in tasks such as re-ranking. Despite this potential overlap, little research has explored reusing or adapting established re-ranking methods to predict relevance judgments, leading to potential resource waste and redundant development. To bridge this gap, we reproduce re-rankers in a re-ranker-as-relevance-judge setup. We design two adaptation strategies: (i) using binary tokens (e.g., "true" and "false") generated by a re-ranker as direct judgments, and (ii) converting continuous re-ranking scores into binary labels via thresholding. We perform extensive experiments on TREC-DL 2019 to 2023 with 8 re-rankers from 3 families, ranging from 220M to 32B, and analyse the evaluation bias exhibited by re-ranker-based judges. Results show that re-ranker-based relevance judges, under both strategies, can outperform UMBRELA, a state-of-the-art LLM-based relevance judge, in around 40% to 50% of the cases; they also exhibit strong self-preference towards their own and same-family re-rankers, as well as cross-family bias.Keywords
All Related Versions
This publication has 29 references indexed in Scilit:
- The Effect of Document Summarization on LLM-Based Relevance Judgments2026
- The landscape of data reuse in interactive information retrieval: Motivations, sources, and evaluation of reusabilityJournal of the Association for Information Science and Technology, 2025
- Context Example Selection for LLM Generated Relevance Assessments2025
- Guiding Retrieval Using LLM-Based Listwise Rankers2025
- Improving the Reusability of Conversational Search Test Collections2025
- AI Can Be Cognitively Biased: An Exploratory Study on Threshold Priming in LLM-Based Batch Relevance Assessment2024
- Large Language Models can Accurately Predict Searcher Preferences2024
- Fine-Tuning LLaMA for Multi-Stage Text Retrieval2024
- Ranked List Truncation for Large Language Model-based Re-Ranking2024
- One-Shot Labeling for Automatic Relevance Estimation2023