Multicentre external validation of a commercial artificial intelligence software to analyse chest radiographs in health screening environments with low disease prevalence
Authors:
- 10 January 2023
- journal article
- research article
- Published by Springer Nature in European Radiology
- Vol. 33 (5) , 3501-3509
Abstract
Objectives To externally validate the performance of a commercial AI software program for interpreting CXRs in a large, consecutive, real-world cohort from primary healthcare centres. Methods A total of 3047 CXRs were collected from two primary healthcare centres, characterised by low disease prevalence, between January and December 2018. All CXRs were labelled as normal or abnormal according to CT findings. Four radiology residents read all CXRs twice with and without AI assistance. The performances of the AI and readers with and without AI assistance were measured in terms of area under the receiver operating characteristic curve (AUROC), sensitivity, and specificity. Results The prevalence of clinically significant lesions was 2.2% (68 of 3047). The AUROC, sensitivity, and specificity of the AI were 0.648 (95% confidence interval [CI] 0.630–0.665), 35.3% (CI, 24.7–47.8), and 94.2% (CI, 93.3–95.0), respectively. AI detected 12 of 41 pneumonia, 3 of 5 tuberculosis, and 9 of 22 tumours. AI-undetected lesions tended to be smaller than true-positive lesions. The readers’ AUROCs ranged from 0.534–0.676 without AI and 0.571–0.688 with AI (all p values < 0.05). For all readers, the mean reading time was 2.96–10.27 s longer with AI assistance (all p values < 0.05). Conclusions The performance of commercial AI in these high-volume, low-prevalence settings was poorer than expected, although it modestly boosted the performance of less-experienced readers. The technical prowess of AI demonstrated in experimental settings and approved by regulatory bodies may not directly translate to real-world practice, especially where the demand for AI assistance is highest. Key Points • This study shows the limited applicability of commercial AI software for detecting abnormalities in CXRs in a health screening population. • When using AI software in a specific clinical setting that differs from the training setting, it is necessary to adjust the threshold or perform additional training with such data that reflects this environment well. • Prospective test accuracy studies, randomised controlled trials, or cohort studies are needed to examine AI software to be implemented in real clinical practice.Keywords
Funding Information
- DongKook Life Science Co
- Lunit Inc.
- department of radiology of Korea university medical center
This publication has 30 references indexed in Scilit:
- Clinical Validation of a Deep Learning Algorithm for Detection of Pneumonia on Chest Radiographs in Emergency Department Patients with Acute Febrile Respiratory IllnessJournal of Clinical Medicine, 2020
- Deep learning algorithm for surveillance of pneumothorax after lung biopsy: a multicenter diagnostic cohort studyEuropean Radiology, 2020
- Deep Learning for Chest Radiograph Diagnosis in the Emergency DepartmentRadiology, 2019
- Deep learning-based detection system for multiclass lesions on chest radiographs: comparison with observer readingsEuropean Radiology, 2019
- Using artificial intelligence to read chest radiographs for tuberculosis detection: A multi-site evaluation of the diagnostic accuracy of three deep learning systemsScientific Reports, 2019
- Development and Validation of a Deep Learning–Based Automated Detection Algorithm for Major Thoracic Diseases on Chest RadiographsJAMA Network Open, 2019
- Diagnostic Case-Control versus Diagnostic Cohort Studies for Clinical Validation of Artificial Intelligence Algorithm PerformanceRadiology, 2019
- Development and Validation of Deep Learning–based Automatic Detection Algorithm for Malignant Pulmonary Nodules on Chest RadiographsRadiology, 2019
- Development and Validation of a Deep Learning–based Automatic Detection Algorithm for Active Pulmonary Tuberculosis on Chest RadiographsClinical Infectious Diseases, 2018
- Development and Validation of a Deep Learning System for Diabetic Retinopathy and Related Eye Diseases Using Retinal Images From Multiethnic Populations With DiabetesJAMA, 2017