Statistical Methods for Assessing Diagnostic Agreement
Open Access
- 4 September 2026
- journal article
- research article
- Published by MDPI AG in Applied Sciences
- Vol. 16 (17) , 8808
Abstract
With the rise of artificial intelligence (AI), an increasing number of AI-based diagnostic tools are being developed. Before clinical implementation, these tools must be validated against existing gold standards. This requires trials that quantify the agreement between AI predictions and reference measurements. However, designing such agreement studies poses methodological challenges that differ substantially from classical superiority trials. This paper aims to provide statistical methods for assessing diagnostic agreement. Methods were categorized according to the measurement scale of the data (nominal, ordinal, continuous)—with a separate group for methods that apply across several scales—and according to the number of raters or measurements involved. A decision tree is provided as a simplified educational framework for method selection rather than as a general method-selection algorithm: design features such as repeated measurements, clustering, spectrum effects, dependence between raters, and an imperfect reference method are not encoded in it and are discussed separately, together with the circularity and confounding issues specific to the validation of AI-based tools. For each method, we summarized assumptions, appropriate use cases, interpretation of results, and available open-source software for sample size calculation and analysis, and we illustrate the sample size calculations in three fully worked examples covering binary, ordinal, and continuous outcomes. We further distinguish conditional inference about one fixed, frozen model version from the broader generalization to a class of algorithms or to future model versions, which require additional sources of algorithmic and dataset variability to be represented in the design and analysis. We conclude by outlining open methodological questions—including Bayesian approaches to agreement estimation, methods for complex AI outputs, agreement models for clustered and repeated-measures designs, and the limited software support for Gwet’s AC1/AC2 sample size planning—that warrant further work as diagnostic technologies and statistical methodology continue to evolve.Keywords
This publication has 53 references indexed in Scilit:
- Statistical Analysis with Missing Data, Third Edition2019
- Guidelines for the Content of Statistical Analysis Plans in Clinical TrialsJAMA, 2017
- Development and Validation of a Deep Learning System for Diabetic Retinopathy and Related Eye Diseases Using Retinal Images From Multiethnic Populations With DiabetesJAMA, 2017
- Sample size formulas for estimating intraclass correlation coefficients with precision and assuranceStatistics in Medicine, 2012
- Variance Estimation of Nominal-Scale Inter-Rater Reliability with Random Selection of RatersPsychometrika, 2008
- Statistical Methods for Rates and Proportions2003
- Sample size requirements for estimating intraclass correlations with desired precisionStatistics in Medicine, 2002
- Sample size and optimal designs for reliability studiesStatistics in Medicine, 1998
- A goodness‐of‐fit approach to inference procedures for the kappa statistic: Confidence interval construction, significance‐testing and sample size estimationStatistics in Medicine, 1992
- High agreement but low kappa: II. Resolving the paradoxesJournal of Clinical Epidemiology, 1990