Do Before You Judge: Self-Reference as a Pathway to Better LLM Evaluation
Authors:
- Wei-Hsiang LinWLWei-Hsiang Lin
- Sheng-Lun WeiSWSheng-Lun Wei
- Hen-Hsen HuangHHHen-Hsen Huang
- Hsin-Hsi ChenHCHsin-Hsi Chen
Preprint
Abstract
LLM-as-Judge frameworks are increasingly popular for AI evaluation, yet research findings on the relationship between models' generation and judgment abilities remain inconsistent. We investigate this relationship through systematic dataset- and instance-level analyses across 11 models and 21 diverse tasks. Despite both capabilities relying on the same underlying knowledge, our analyses reveal they are only weakly correlated, primarily due to LLMs' sensitivity to the responses being judged. To address this, we propose a self-reference-guided evaluation strategy that leverages a model's own answers as references. This approach significantly strengthens the correlation between generation and judgment abilities, offering a practical path to align these skills and providing a reliable proxy for model selection in evaluation tasks.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- LawBench: Benchmarking Legal Knowledge of Large Language Models2024
- Humans or LLMs as the Judge? A Study on Judgement Bias2024
- Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment2024
- Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models2024
- Benchmarking Cognitive Biases in Large Language Models as Evaluators2024
- ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs2024
- Large Language Models are not Fair Evaluators2024
- Aligning Large Language Models by On-Policy Self-Judgment2024
- OceanGPT: A Large Language Model for Ocean Science Tasks2024
- LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models2023