Evaluating Generalizability of Deep Learning Models Using Indian-COVID-19 CT Dataset
Preprint
Abstract
Computer tomography (CT) have been routinely used for the diagnosis of lung diseases and recently, during the pandemic, for detecting the infectivity and severity of COVID-19 disease. One of the major concerns in using ma-chine learning (ML) approaches for automatic processing of CT scan images in clinical setting is that these methods are trained on limited and biased sub-sets of publicly available COVID-19 data. This has raised concerns regarding the generalizability of these models on external datasets, not seen by the model during training. To address some of these issues, in this work CT scan images from confirmed COVID-19 data obtained from one of the largest public repositories, COVIDx CT 2A were used for training and internal vali-dation of machine learning models. For the external validation we generated Indian-COVID-19 CT dataset, an open-source repository containing 3D CT volumes and 12096 chest CT images from 288 COVID-19 patients from In-dia. Comparative performance evaluation of four state-of-the-art machine learning models, viz., a lightweight convolutional neural network (CNN), and three other CNN based deep learning (DL) models such as VGG-16, ResNet-50 and Inception-v3 in classifying CT images into three classes, viz., normal, non-covid pneumonia, and COVID-19 is carried out on these two datasets. Our analysis showed that the performance of all the models is comparable on the hold-out COVIDx CT 2A test set with 90% - 99% accuracies (96% for CNN), while on the external Indian-COVID-19 CT dataset a drop in the performance is observed for all the models (8% - 19%). The traditional ma-chine learning model, CNN performed the best on the external dataset (accu-racy 88%) in comparison to the deep learning models, indicating that a light-weight CNN is better generalizable on unseen data. The data and code are made available at https://github.com/aleesuss/c19.Keywords
All Related Versions
This publication has 27 references indexed in Scilit:
- A fully automated deep learning-based network for detecting COVID-19 from a new and large lung CT scan datasetBiomedical Signal Processing and Control, 2021
- Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scansNature Machine Intelligence, 2021
- A deep learning algorithm using CT images to screen for Corona virus disease (COVID-19)European Radiology, 2021
- External validation demonstrates limited clinical utility of the interpretable mortality prediction model for patients with COVID-19Nature Machine Intelligence, 2020
- Clinically Applicable AI System for Accurate Diagnosis, Quantitative Measurements, and Prognosis of COVID-19 Pneumonia Using Computed TomographyCell, 2020
- Application of deep learning technique to manage COVID-19 in routine clinical practice using CT images: Results of 10 convolutional neural networksComputers in Biology and Medicine, 2020
- Artificial intelligence–enabled rapid diagnosis of patients with COVID-19Nature Medicine, 2020
- COVID-19 pneumonia: A review of typical CT findings and differential diagnosisDiagnostic and Interventional Imaging, 2020
- Diagnosis of the Coronavirus disease (COVID-19): rRT-PCR or CT?European Journal of Radiology, 2020
- ImageNet: A large-scale hierarchical image database2009