Classification performance bias between training and test sets in a limited mammography dataset
Open Access
- 7 February 2024
- journal article
- research article
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 19 (2) , e0282402
Abstract
Objectives: To assess the performance bias caused by sampling data into training and test sets in a mammography radiomics study.Methods: Mammograms from 700 women were used to study upstaging of ductal carcinoma in situ. The dataset was repeatedly shuffled and split into training (n = 400) and test cases (n = 300) forty times. For each split, cross-validation was used for training, followed by an assessment of the test set. Logistic regression with regularization and support vector machine were used as the machine learning classifiers. For each split and classifier type, multiple models were created based on radiomics and/or clinical features.Results: Area under the curve (AUC) performances varied considerably across the different data splits (e.g., radiomics regression model: train 0.58–0.70, test 0.59–0.73). Performances for regression models showed a tradeoff where better training led to worse testing and vice versa. Cross-validation over all cases reduced this variability, but required samples of 500+ cases to yield representative estimates of performance.Conclusions: In medical imaging, clinical datasets are often limited to relatively small size. Models built from different training sets may not be representative of the whole dataset. Depending on the selected data split and model, performance bias could lead to inappropriate conclusions that might influence the clinical significance of the findings.Advances in knowledge: Performance bias can result from model testing when using limited datasets. Optimal strategies for test set selection should be developed to ensure study conclusions are appropriate.Keywords
All Related Versions
Funding Information
- National Cancer Institute (U01-CA214183)
- National Cancer Institute (R01-CA185138)
- DOD Breast Cancer Research Program (W81XWH-14-1-0473)
- Breast Cancer Research Foundation (BCRF-16-183)
- Breast Cancer Research Foundation (BCRF-17-073)
- Cancer Research UK and Dutch Cancer Society (C38317/A24043)
This publication has 25 references indexed in Scilit:
- Cross-validation failure: Small sample sizes lead to large error barsNeuroImage, 2018
- On Splitting Training and Validation Set: A Comparative Study of Cross-Validation, Bootstrap and Systematic Sampling for Estimating the Generalization Performance of Supervised LearningJournal of Analysis and Testing, 2018
- Prediction of Occult Invasive Disease in Ductal Carcinoma in Situ Using Deep Learning FeaturesJournal of the American College of Radiology, 2018
- Surgical Upstaging Rates for Vacuum Assisted Biopsy Proven DCIS: Implications for Active Surveillance TrialsAnnals of Surgical Oncology, 2017
- ChestX-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases2017
- Dermatologist-level classification of skin cancer with deep neural networksNature, 2017
- Large scale deep learning for computer aided detection of mammographic lesionsMedical Image Analysis, 2017
- Long-term outcome of DCIS patients: p53 as a biomarker of ipsilateral recurrence.Journal of Clinical Oncology, 2011
- Ductal Carcinoma in Situ at Core-Needle Biopsy: Meta-Analysis of Underestimation and Predictors of Invasive Breast CancerRadiology, 2011
- The natural history of low‐grade ductal carcinoma in situ of the breast in women treated by biopsy only revealed over 30 years of long‐term follow‐upCancer, 2005