Haiko Schurz

AI cancer detection models require calibration to attain the desired balance between cancer detection rate (CDR) and false positive rate. In this study, we simulate the impact of six types of mismatches between the calibration population and the clinical target population, by creating purposefully non-representative datasets to calibrate AI for clinical settings. Mismatching the acquisition year between healthy and cancer-diagnosed screening participants led to a distortion in CDR between −3% to +19%. Mismatching age led to a distortion in CDR between −0.2% to +27%. Mismatching breast density distribution led to a distortion in CDR between +1% to 16%. Mismatching mammography vendors lead to a distortion in CDR between −32% to + 33%. Mismatches between calibration population and target clinical population lead to clinically important deviations. It is vital for safe clinical AI integration to ensure that important aspects of the calibration population are representative of the target population.

Machine learning (ML) is rapidly progressing in many scientific fields, including medicine.¹ For breast cancer, we are beholding the emergence of commercially available artificial intelligence (AI) systems for breast cancer detection (AI-CADe),^{2 – 8} triaging, diagnosis, and risk assessment.⁹ If the performance of such systems proves to be accurate and robust in a clinical setting, incorporating them into the screening process can significantly benefit both the hospital and the screening participants. The most common use case, AI-CADe, would reduce radiologist workload and potentially reduce the rate of missed cancers.¹⁰ Some challenges remain despite the potential impact of adopting these technologies in the screening process. Collectively these challenges are known as the AI translational gap, which is currently under special investigation by the Federal Drug Administration in the United States of America (USA).¹¹