Source Matters: Source Dataset Impact on Model Robustness in Medical Imaging
Preprint
Abstract
Transfer learning has become an essential part of medical imaging classification algorithms, often leveraging ImageNet weights. However, the domain shift from natural to medical images has prompted alternatives such as RadImageNet, often demonstrating comparable classification performance. However, it remains unclear whether the performance gains from transfer learning stem from improved generalization or shortcut learning. To address this, we investigate potential confounders -- whether synthetic or sampled from the data -- across two publicly available chest X-ray and CT datasets. We show that ImageNet and RadImageNet achieve comparable classification performance, yet ImageNet is much more prone to overfitting to confounders. We recommend that researchers using ImageNet-pretrained models reexamine their model robustness by conducting similar experiments. Our code and experiments are available at https://github.com/DovileDo/source-matters.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- “Shortcuts” Causing Bias in Radiology Artificial Intelligence: Causes, Evaluation, and MitigationJournal of the American College of Radiology, 2023
- RadImageNet: An Open Radiologic Deep Learning Research Dataset for Effective Transfer LearningRadiology: Artificial Intelligence, 2022
- Noise reduction profile: A new method for evaluation of noise reduction techniques in CTMedical Physics, 2021
- Deep learning for distinguishing normal versus abnormal chest radiographs and generalization to two unseen diseases tuberculosis and COVID-19Scientific Reports, 2021
- Physics-based Noise Modeling for Extreme Low-light PhotographyIEEE Transactions on Pattern Analysis and Machine Intelligence, 2021
- Rethinking pre-training on medical imagingJournal of Visual Communication and Image Representation, 2021
- Do no harm: a roadmap for responsible machine learning for health careNature Medicine, 2019
- Cats or CAT scans: Transfer learning from natural or medical image source data sets?Current Opinion in Biomedical Engineering, 2019
- Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional studyPLOS Medicine, 2018