Abstract
LLM-as-a-judge has emerged as a promising strategy for evaluating recommender systems. Yet, treating simulated ratings as real feedback can make unreliable evaluation appear reliable. This risk is especially consequential in sensitive domains such as nutrition and health. In this paper, we treat the LLM-as-a-judge paradigm as a calibrated relevance simulation problem: an LLM-based simulator predicts human preferences from user and item representations, and conformal prediction converts them into uncertainty intervals. To account for the nature of recommendation, we introduce stratified conformal calibration by user rating variance. Experiments under three LLMs and seven conformal prediction methods show that point estimates of preferences hide failures on low-rating items, while conformal interval width is driven by user-level heterogeneity. Source code and data: https://github.com/maddalena-amendola/conformal_prediction_recsys.
Funding Information
  • Horizon Europe (B53C24011590001)
  • COST Action KGELL (CA24121)