An information-oriented paradigm in evaluating accuracy and agreement in radiology
Open Access
- 20 March 2023
- journal article
- research article
- Published by Springer Nature in European Radiology Experimental
- Vol. 7 (1) , 1-7
Abstract
The goal of any radiological diagnostic process is to gain information about the patient’s status. However, the mathematical notion of information is usually not adopted to measure the performance of a diagnostic test or the agreement among readers in providing a certain diagnosis. Indeed, commonly used metrics for assessing diagnostic accuracy (e.g., sensitivity and specificity) or inter-reader agreement (Cohen $$\kappa$$ statistics) use confusion matrices containing the number of true- and false positives/negatives results of a test, or the number of concordant/discordant categorizations, respectively, thus lacking proper information content. We present a methodological paradigm, based on Shannon’s information theory, aiming to measure both accuracy and agreement in diagnostic radiology. This approach models the information flow as a “diagnostic channel” connecting the state of the patient’s disease and the radiologist or, in the case of agreement analysis, as an “agreement channel” linking two or more radiologists evaluating the same set of images. For both cases, we proposed some measures, derived from Shannon’s mutual information, which can represent an alternative way to express diagnostic accuracy and agreement in radiology.Key points• Diagnostic processes can be modeled with information theory (IT).• IT metrics of diagnostic accuracy are independent from disease prevalence.• IT metrics of inter-reader agreements can overcome Cohen κ pitfalls.
Keywords
This publication has 21 references indexed in Scilit:
- Informational analysis: a Shannon theoretic approach to measure the performance of a diagnostic testMedical & Biological Engineering & Computing, 2015
- Clinicians are right not to like Cohen'sBMJ, 2013
- The channel capacity of a diagnostic test as a function of test sensitivity and test specificityStatistical Methods in Medical Research, 2012
- The dependence of Cohen's kappa on the prevalence does not matterJournal of Clinical Epidemiology, 2005
- Spectrum Bias or Spectrum Effect? Subgroup Variation in Diagnostic Test EvaluationAnnals of Internal Medicine, 2002
- Fifty years of Shannon theoryIEEE Transactions on Information Theory, 1998
- A rEAPPRAISAL OF THE KAPPA COEFFICIENTJournal of Clinical Epidemiology, 1988
- Problems of Spectrum and Bias in Evaluating the Efficacy of Diagnostic TestsNew England Journal of Medicine, 1978
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960
- A Mathematical Theory of CommunicationBell System Technical Journal, 1948