Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement
Authors:
Preprint
Abstract
We present a principled approach to provide LLM-based evaluation with a rigorous guarantee of human agreement. We first propose that a reliable evaluation method should not uncritically rely on model preferences for pairwise evaluation, but rather assess the confidence of judge models and selectively decide when to trust its judgement. We then show that under this selective evaluation framework, human agreement can be provably guaranteed -- such that the model evaluation aligns with that of humans to a user-specified agreement level. As part of our framework, we also introduce Simulated Annotators, a novel confidence estimation method that significantly improves judge calibration and thus enables high coverage of evaluated instances. Finally, we propose Cascaded Selective Evaluation, where we use cheaper models as initial judges and escalate to stronger models only when necessary -- again, while still providing a provable guarantee of human agreement. Experimental results show that Cascaded Selective Evaluation guarantees strong alignment with humans, far beyond what LLM judges could achieve without selective evaluation. For example, on a subset of Chatbot Arena where GPT-4 almost never achieves 80% human agreement, our method, even while employing substantially cost-effective models such as Mistral-7B, guarantees over 80% human agreement with almost 80% test coverage.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Learn then test: Calibrating predictive algorithms to achieve risk controlThe Annals of Applied Statistics, 2025
- Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models2024
- Benchmarking Cognitive Biases in Large Language Models as Evaluators2024
- Large Language Models are not Fair Evaluators2024
- Dissecting Human and LLM Preferences2024
- G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment2023
- Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback2023
- Can Large Language Models Be an Alternative to Human Evaluations?2023
- AlignScore: Evaluating Factual Consistency with A Unified Alignment Function2023
- Distribution-free, Risk-controlling Prediction SetsJournal of the ACM, 2021