Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry
Authors:
- Yong ZhangYZYong Zhang
- Ming LiMLMing Li
- Yiming ZengYZYiming Zeng
- Ning ChengNCNing Cheng
- Shaojun WangSWShaojun Wang
Preprint
Abstract
Large language models (LLMs) are widely used as reference-free evaluators via prompting, but this "LLM-as-a-Judge" paradigm is costly, opaque, and sensitive to prompt design. In this work, we investigate whether smaller models can serve as efficient evaluators by leveraging internal representations instead of surface generation. We uncover a consistent empirical pattern: small LMs, despite with weak generative ability, encode rich evaluative signals in their hidden states. This motivates us to propose the Semantic Capacity Asymmetry Hypothesis: evaluation requires significantly less semantic capacity than generation and can be grounded in intermediate representations, suggesting that evaluation does not necessarily need to rely on large-scale generative models but can instead leverage latent features from smaller ones. Our findings motivate a paradigm shift from LLM-as-a-Judge to Representation-as-a-Judge, a decoding-free evaluation strategy that probes internal model structure rather than relying on prompted output. We instantiate this paradigm through INSPECTOR, a probing-based framework that predicts aspect-level evaluation scores from small model representations. Experiments on reasoning benchmarks (GSM8K, MATH, GPQA) show that INSPECTOR substantially outperforms prompting-based small LMs and closely approximates full LLM judges, while offering a more efficient, reliable, and interpretable alternative for scalable evaluation.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- LLaMAs Have Feelings Too: Unveiling Sentiment and Emotion Representations in LLaMA Models Through Probing2025
- Holmes ⌕ A Benchmark to Assess the Linguistic Competence of Language ModelsTransactions of the Association for Computational Linguistics, 2024
- State of What Art? A Call for Multi-Prompt LLM EvaluationTransactions of the Association for Computational Linguistics, 2024
- Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translation2024
- SocREval: Large Language Models with the Socratic Method for Reference-free Reasoning Evaluation2024
- Use all tokens method to improve semantic relationship learningExpert Systems with Applications, 2023
- G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment2023
- ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness2023
- Probing LLMs for Joint Encoding of Linguistic Categories2023
- A Primer in BERTology: What We Know About How BERT WorksTransactions of the Association for Computational Linguistics, 2020