Impact of prevalence and case distribution in lab-based diagnostic imaging studies
- 21 January 2019
- journal article
- research article
- Published by SPIE-Intl Soc Optical Eng in Journal of Medical Imaging
- Vol. 6 (01) , 015501-9
Abstract
We investigated effects of prevalence and case distribution on radiologist diagnostic performance as measured by area under the receiver operating characteristic curve (AUC) and sensitivity-specificity in lab-based reader studies evaluating imaging devices. Our retrospective reader studies compared full-field digital mammography (FFDM) to screen-film mammography (SFM) for women with dense breasts. Mammograms were acquired from the prospective Digital Mammographic Imaging Screening Trial. We performed five reader studies that differed in terms of cancer prevalence and the distribution of noncancers. Twenty radiologists participated in each reader study. Using split-plot study designs, we collected recall decisions and multilevel scores from the radiologists for calculating sensitivity, specificity, and AUC. Differences in reader-averaged AUCs slightly favored SFM over FFDM (biggest AUC difference: 0.047, SE = 0.023, p = 0.047), where standard error accounts for reader and case variability. The differences were not significant at a level of 0.01 (0.05/5 reader studies). The differences in sensitivities and specificities were also indeterminate. Prevalence had little effect on AUC (largest difference: 0.02), whereas sensitivity increased and specificity decreased as prevalence increased. We found that AUC is robust to changes in prevalence, while radiologists were more aggressive with recall decisions as prevalence increased.Keywords
This publication has 26 references indexed in Scilit:
- If You Don’t Find It Often, You Often Don’t Find It: Why Some Cancers Are Missed in Breast Cancer ScreeningPLOS ONE, 2013
- A Framework for Random-Effects ROC Analysis: Biases with the Bootstrap and Other Variance EstimatorsCommunications in Statistics - Theory and Methods, 2009
- Comparison of Soft-copy and Hard-copy Reading for Full-Field Digital MammographyRadiology, 2009
- The “Laboratory” Effect: Comparing Radiologists' Performance and Variability during Prospective Clinical and Laboratory Mammography InterpretationsRadiology, 2008
- Accuracy of Soft-Copy Digital Mammography versus That of Screen-Film Mammography according to Digital Manufacturer: ACRIN DMIST Retrospective Multireader StudyRadiology, 2008
- The Prevalence Effect in a Laboratory EnvironmentAcademic Radiology, 2007
- Diagnostic Performance of Digital versus Film Mammography for Breast-Cancer ScreeningNew England Journal of Medicine, 2005
- Rare items often missed in visual searchesNature, 2005
- ACRIN—lessons learned in conducting multi-center trials of imaging and cancerCancer Imaging, 2005
- Optimality of perceptual decision criteria.Journal of Experimental Psychology, 1966