Clinical selectivity and failure modes of automated chest radiograph report evaluation metrics: a cross-dataset analysis of ReXErr-v1 and RadEvalX
Preprint
Abstract
Objectives: To test whether radiology report evaluation metrics distinguish clinically meaningful errors from textual changes and align with radiologist-assessed error burden. Methods: Cross-dataset evaluation used ReXErr-v1 (2,708 report pairs; 5,724 paired error sentences) and 100 RadEvalX report pairs with expert error counts. BLEU-4, ROUGE-L and METEOR were assessed in ReXErr-v1; RadEvalX analyses included these plus BERTScore, CheXbert, RadGraph F1 and RadCliQ. Outcomes were ReXErr-v1 pairwise win rate and AUROC for clinical-content versus linguistic errors, and RadEvalX Spearman correlation with clinically significant error count and AUROC for any significant error. Confidence intervals used 10,000 clustered percentile bootstrap resamples; Holm adjustment-controlled multiplicity. Results: ReXErr-v1 paired-sentence win rates were 0.986 for BLEU-4, 0.999 for ROUGE-L and 0.998 for METEOR, but discrimination of clinical-content from linguistic errors was modest (AUROC 0.609–0.620). Penalty magnitude was strongly associated with textual change after adjustment for error type (normalised character edit distance coefficient 0.746; 95% CI 0.705–0.788; P<0.001). In RadEvalX, CheXbert showed the highest correlation with clinically significant errors (rho=0.413; 95% CI 0.223–0.578) and highest AUROC (0.742; 95% CI 0.638–0.836). Conclusions: Near-ceiling sensitivity to textual corruption did not imply sensitivity to clinical significance. CheXbert showed the highest alignment with expert error assessment, although pairwise superiority was not demonstrated over all comparators and performance remained moderate. Advances in knowledge: This study separates textual corruption sensitivity from clinical selectivity across complementary synthetic and expert-annotated benchmarks, showing that apparent error-detection performance can be strongly influenced by textual change rather than clinical importance alone.Keywords
This publication has 15 references indexed in Scilit:
- GEMA-Score: Granular Explainable Multi-Agent Scoring Framework for Radiology Report EvaluationProceedings of the AAAI Conference on Artificial Intelligence, 2026
- ReXErr: Synthesizing Clinically Meaningful Errors in Diagnostic Radiology Reports2024
- GREEN: Generative Radiology Report Evaluation and Error Notation2024
- RaTEScore: A Metric for Radiology Report Generation2024
- Evaluating progress in automatic chest X-ray radiology report generationPatterns, 2023
- Array programming with NumPyNature, 2020
- SciPy 1.0: fundamental algorithms for scientific computing in PythonNature Methods, 2020
- Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT2020
- Preparing a collection of radiology examinations for distribution and retrievalJournal of the American Medical Informatics Association, 2015
- Longitudinal data analysis using generalized linear modelsBiometrika, 1986