Reliability and construct validity of a multi-model LLM-as-judge evaluator battery for clinician-facing clinical question answering: a controlled adversarial benchmark study (Preprint)
Author:
Preprint
- 17 September 2026
- preprint
- Published by JMIR Publications Inc. in JMIR Preprints
Abstract
BACKGROUND Open-ended clinical question systems generate fluent free-text for which conventional accuracy metrics are insufficient. Complex questions may admit more than one defensible answer depending on clinical context, jurisdiction, guideline version and the intent of the person asking. Human expert review remains important but is costly, hard to scale and variable. Large language models (LLMs) are increasingly used as automated evaluators, but an LLM-as-judge should be characterised as a measurement instrument before its outputs are treated as evidence. OBJECTIVE To characterise the inter-judge reliability, within-judge stability, severity-rating heterogeneity, difficulty discrimination and construct validity of an LLM-as-judge battery used to evaluate clinician-facing clinical question answering. METHODS We evaluated a three-model FACTFULNESS ensemble (Claude Opus 4.7, Gemini 3.1 Pro, GPT-5.4, each with native web search) and a single-model COMPLETENESS evaluator (GPT-5.4) on an author-constructed adversarial bank of 1,424 clinical questions across five difficulty tiers and 16 construct families. Of 1,421 scored items, 1,173 (82.5%) carried complete three-judge verdicts. Agreement on binary factuality used Gwet AC1, Fleiss kappa and unanimity, overall and within 22 strata. Sensitivity analyses addressed the complete-case restriction, claim-set non-identity, repeated scoring and retrieved-evidence overlap. RESULTS Gwet AC1 was 0.344 (95% CI 0.302-0.386), Fleiss kappa 0.212, and 46.3% of items unanimous. AC1 fell from 0.60 at difficulty tier D1 to 0.27 at D5, ranged from 0.79 to 0.13 across construct families, and ranged from 0.183 to 0.525 between judge pairs with non-overlapping intervals. The serious-or-worse harm-severity rate differed 17.1-fold across judges (0.23%, 2.34% and 3.94%). Requiring two of three judges reduced HIGH+ classification from 13.79% to 2.89%, and no answer met CRITICAL by consensus (0/1,421). FACTFULNESS-target families were more often consensus-inaccurate than non-target families (36.8% vs 27.9%, P=0.002), whereas the COMPLETENESS check was null once refusal-driven security items were excluded (1.7% vs 4.3%, P=0.27). On rescoring byte-identical answers, mean within-judge AC1 was 0.703 against 0.265 between judges, and the consensus risk band was reproduced across all five replicates for 52.0% of items. CONCLUSIONS Treated as a measurement instrument, the battery's reliability and severity grading depended strongly on judge identity and on the task being judged, and one of two planned construct checks behaved as intended. Consensus reduced extreme classifications, and aggregate estimates were reproducible enough to support period-on-period monitoring, whereas single-run item-level verdicts were not. Multi-model batteries are therefore usable for monitoring when consensus is reported with judge-specific dispersion, missingness and stratified reliability. Human-anchored calibration remains necessary before severity scale or risk labels can be interpreted as absolute clinical risk. CLINICALTRIAL NAKeywords
This publication has 36 references indexed in Scilit:
- Guideline-anchored retrieval-augmented generation outperforms baseline and literature-only configurations in gynecologic oncology decision support: A pre-integration benchmarkGynecologic Oncology, 2026
- Evaluating the safety of large language models in healthcare and dentistry: adversarial testing approachesBDJ Open, 2026
- Vulnerability of Large Language Models to Prompt Injection When Providing Medical AdviceJAMA Network Open, 2025
- The TRIPOD-LLM reporting guideline for studies using large language modelsNature Medicine, 2025
- Evaluation of ChatGPT-generated medical responses: A systematic review and meta-analysisJournal of Biomedical Informatics, 2024
- Large language models encode clinical knowledgeNature, 2023
- Gwet's AC1 is not a substitute for Cohen's kappa – A comparison of basic propertiesMethodsX, 2023
- Industry sponsorship and research outcomeCochrane Database of Systematic Reviews, 2017
- Two-sided confidence intervals for the single proportion: comparison of seven methodsStatistics in Medicine, 1998
- High agreement but low Kappa: I. the problems of two paradoxesJournal of Clinical Epidemiology, 1990