Validating the accuracy of deep learning for the diagnosis of pneumonia on chest x-ray against a robust multimodal reference diagnosis: a post hoc analysis of two prospective studies
Open Access
- 2 February 2024
- journal article
- research article
- Published by Springer Nature in European Radiology Experimental
- Vol. 8 (1) , 1-10
Abstract
Background: Artificial intelligence (AI) seems promising in diagnosing pneumonia on chest x-rays (CXR), but deep learning (DL) algorithms have primarily been compared with radiologists, whose diagnosis can be not completely accurate. Therefore, we evaluated the accuracy of DL in diagnosing pneumonia on CXR using a more robust reference diagnosis. Methods: We trained a DL convolutional neural network model to diagnose pneumonia and evaluated its accuracy in two prospective pneumonia cohorts including 430 patients, for whom the reference diagnosis was determined a posteriori by a multidisciplinary expert panel using multimodal data. The performance of the DL model was compared with that of senior radiologists and emergency physicians reviewing CXRs and that of radiologists reviewing computed tomography (CT) performed concomitantly. Results: Radiologists and DL showed a similar accuracy on CXR for both cohorts (p ≥ 0.269): cohort 1, radiologist 1 75.5% (95% confidence interval 69.1–80.9), radiologist 2 71.0% (64.4–76.8), DL 71.0% (64.4–76.8); cohort 2, radiologist 70.9% (64.7–76.4), DL 72.6% (66.5–78.0). The accuracy of radiologists and DL was significantly higher (p ≤ 0.022) than that of emergency physicians (cohort 1 64.0% [57.1–70.3], cohort 2 63.0% [55.6–69.0]). Accuracy was significantly higher for CT (cohort 1 79.0% [72.8–84.1], cohort 2 89.6% [84.9–92.9]) than for CXR readers including radiologists, clinicians, and DL (all p-values < 0.001). Conclusions: When compared with a robust reference diagnosis, the performance of AI models to identify pneumonia on CXRs was inferior than previously reported but similar to that of radiologists and better than that of emergency physicians. Relevance statement: The clinical relevance of AI models for pneumonia diagnosis may have been overestimated. AI models should be benchmarked against robust reference multimodal diagnosis to avoid overestimating its performance. Trial registration: NCT02467192, and NCT01574066. Key point: • We evaluated an openly-access convolutional neural network (CNN) model to diagnose pneumonia on CXRs. • CNN was validated against a strong multimodal reference diagnosis. • In our study, the CNN performance (area under the receiver operating characteristics curve 0.74) was lower than that previously reported when validated against radiologists’ diagnosis (0.99 in a recent meta-analysis). • The CNN performance was significantly higher than emergency physicians’ (p ≤ 0.022) and comparable to that of board-certified radiologists (p ≥ 0.269). GraphicalKeywords
Funding Information
- Research and Development Grant of the Geneva University Hospital (PRD 7-2015-II)
- Research and Development Grant of the Geneva University Hospital (PRD 11-2017-II)
- Ligue Pulmonaire Genevoise
This publication has 37 references indexed in Scilit:
- Exhaled breath condensate in intubated neonates- a window into the lung’s glutathione statusRespiratory Research, 2014
- High discordance of chest x-ray and computed tomography for detection of pulmonary opacities in ED patients: implications for diagnosing pneumoniaThe American Journal of Emergency Medicine, 2013
- Guidelines for the management of adult lower respiratory tract infections - Full versionClinical Microbiology and Infection, 2011
- Interobserver Reliability of Radiologists’ Interpretations of Mobile Chest Radiographs for Nursing Home–Acquired PneumoniaJournal of the American Medical Directors Association, 2006
- Inter-observer variation in the interpretation of chest radiographs for pneumonia in community-acquired lower respiratory tract infectionsClinical Radiology, 2004
- Pneumonia in the very oldThe Lancet Infectious Diseases, 2004
- High‐Resolution Computed Tomography for the Diagnosis of Community‐Acquired PneumoniaClinical Infectious Diseases, 1998
- Interobserver Reliability of the Chest Radiograph in Community-Acquired PneumoniaChest, 1996
- Interobserver Variability in the Interpretation of Chest Roentgenograms of Patients With Possible PneumoniaArchives of internal medicine (1960), 1994
- Probable Inference, the Law of Succession, and Statistical InferenceJournal of the American Statistical Association, 1927