A Multi-Reader Multi-Case Study Comparing Mammographic Breast Cancer Detection Rates on Mammograms by Resident and Consultant Radiologists in Singapore with and Without the Use of AI Assistance
Authors:
Preprint
- 21 December 2023
- preprint
- Published by Elsevier in SSRN Electronic Journal
Abstract
Introduction: Breast cancer is the most common type of cancer in women globally and mammograms are a primary method for diagnosing it. The challenge of interpreting mammograms is exacerbated by a shortage of dedicated breast radiologists and the lengthy training required to cultivate such expertise. This study is designed to investigate the performance of AI assistance for resident radiologists compared to consultant radiologists and to conduct a cost analysis of implementing AI in a diagnostic setting. Methods: A multi-reader multi-case study was conducted at the National University Hospital of Singapore from May to August 2023. De-identified mammograms from 550 women formed the dataset. Seventeen radiologists read mammograms for six weeks without and with AI assistance, with a one-month washout period in between. FxMammo AI software provided heatmaps over areas of suspicion and malignancy risk scores. Diagnostic performance between resident and consultant radiologists were compared with and without AI assistance. Cost analysis was performed. Findings: A total of 550 women were included. 261 had breast cancer whilst 289 had normal or benign breast lesions. Majority had dense breasts (66.9%). The diagnostic performance of AI standalone versus consultant radiologists was comparable AUROC of 0.93 (95% CI 0.90-0.95) versus 0.90 (95% CI 0.89-0.92) (p=0.97). With AI assistance, both junior and senior residents showed improvement in diagnostic performance AUROC 0.84 to 0.86 (95% CI 0.85- 0.87), p=0.001, and AUROC 0.85 to 0.89 (95% CI 0.87- 0.90) respectively, p<0.001). With AI assistance, the AUROC of senior residents was comparable to consultant radiologists with difference in AUROC of 0.02 (95% CI -0.001 – 0.04, p=0.055). Time savings were observed, particularly in non-malignant cases, with potential annual cost savings of SGD $2,732,104 - $2,766,193. Interpretation: The AI performance was comparable to consultant radiologists in a study population with high proportion of dense breasts. The findings suggest that AI can bridge gaps in expertise between senior resident radiologists and consultant radiologists. Funding: This study was supported by Temasek Foundation and Ministry of Health Singapore through the CHI START UP ENTERPRISE LINK (CHISEL) programme organized by Centre for Healthcare Innovation in 2022, NUHS Clinician Scientist Program 2.0 (NCSP 2.0) under Department of Surgery, National University Hospital, and Breast Cancer Screening Prevention Programme (NUHSRO/2020/121/BCSPP/LOA) under Yong Loo Lin School of Medicine, National University of Singapore. The grant was used to fund the manpower required for the project. The study design, data collection, data analysis and interpretation were conducted by the study team independently. Declaration of Interest: Mengling Fang, As a faculty member at NUHS and a co-founder of FathomX, I have a financial and professional interest in the success and adoption of FathomX's AI technology in the field of radiology. This interest may be perceived as a competing interest, as the outcomes of the research could have implications for the acceptance and utilization of AI technology in radiology. All other have nothing to declare. Ethical Approval: The research protocol received approval from the National Health Group Institutional Review Boards on 16 December 2022 (Reference Number 2022/00843).Keywords
This publication has 22 references indexed in Scilit:
- Preventing Physician Burnout in Breast Imaging: Scope of the Problem and Keys to SuccessCurrent Problems in Diagnostic Radiology, 2021
- Identifying normal mammograms in a large screening population using artificial intelligenceEuropean Radiology, 2020
- Artificial Intelligence: A Primer for Breast Imaging RadiologistsJournal of Breast Imaging, 2020
- Key challenges for delivering clinical impact with artificial intelligenceBMC Medicine, 2019
- Detection of Breast Cancer with Mammography: Effect of an Artificial Intelligence Support SystemRadiology, 2019
- Toward Augmented RadiologistsAcademic Radiology, 2018
- Consequences of False-Positive Screening MammogramsJAMA Internal Medicine, 2014
- Individual and Combined Effects of Age, Breast Density, and Hormone Replacement Therapy Use on the Accuracy of Screening MammographyAnnals of Internal Medicine, 2003
- Ethnic differences in mammographic densitiesInternational Journal of Epidemiology, 2001
- Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric ApproachBiometrics, 1988