Conformal Prediction for Statistically Reliable LLM-as-a-Judge in Recommender Systems
Open Access
- 6 November 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 6844-6849
Abstract
LLM-as-a-judge has emerged as a promising strategy for evaluating recommender systems. Yet, treating simulated ratings as real feedback can make unreliable evaluation appear reliable. This risk is especially consequential in sensitive domains such as nutrition and health. In this paper, we treat the LLM-as-a-judge paradigm as a calibrated relevance simulation problem: an LLM-based simulator predicts human preferences from user and item representations, and conformal prediction converts them into uncertainty intervals. To account for the nature of recommendation, we introduce stratified conformal calibration by user rating variance. Experiments under three LLMs and seven conformal prediction methods show that point estimates of preferences hide failures on low-rating items, while conformal interval width is driven by user-level heterogeneity. Source code and data: https://github.com/maddalena-amendola/conformal_prediction_recsys.Keywords
Funding Information
- Horizon Europe (B53C24011590001)
- COST Action KGELL (CA24121)
This publication has 9 references indexed in Scilit:
- Evaluating Podcast Recommendations with Profile-Aware LLM-as-a-Judge2025
- How Can Recommender Systems Benefit from Large Language Models: A SurveyACM Transactions on Information Systems, 2025
- Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction2025
- Large Language Models can Accurately Predict Searcher Preferences2024
- Can LLM be a Personalized Judge?2024
- GPTScore: Evaluate as You Desire2024
- HUMMUS: A Linked, Healthiness-Aware, User-centered and Argument-Enabling Recipe Data Set for Recommendation2023
- G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment2023
- A comparison of some conformal quantile regression methodsStat, 2020