Algorithmic transparency and interpretability measures improve radiologists’ performance in BI-RADS 4 classification
Open Access
- 25 October 2022
- journal article
- research article
- Published by Springer Nature in European Radiology
- Vol. 33 (3) , 1844-1851
Abstract
Objective: To evaluate the perception of different types of AI-based assistance and the interaction of radiologists with the algorithm’s predictions and certainty measures. Methods: In this retrospective observer study, four radiologists were asked to classify Breast Imaging-Reporting and Data System 4 (BI-RADS4) lesions (n = 101 benign, n = 99 malignant). The effect of different types of AI-based assistance (occlusion-based interpretability map, classification, and certainty) on the radiologists’ performance (sensitivity, specificity, questionnaire) were measured. The influence of the Big Five personality traits was analyzed using the Pearson correlation. Results: Diagnostic accuracy was significantly improved by AI-based assistance (an increase of 2.8% ± 2.3%, 95 %-CI 1.5 to 4.0 %, p = 0.045) and trust in the algorithm was generated primarily by the certainty of the prediction (100% of participants). Different human-AI interactions were observed ranging from nearly no interaction to humanization of the algorithm. High scores in neuroticism were correlated with higher persuasibility (Pearson’s r = 0.98, p = 0.02), while higher consciousness and change of accuracy showed an inverse correlation (Pearson’s r = −0.96, p = 0.04). Conclusion: Trust in the algorithm’s performance was mostly dependent on the certainty of the predictions in combination with a plausible heatmap. Human-AI interaction varied widely and was influenced by personality traits. Key Points: • AI-based assistance significantly improved the diagnostic accuracy of radiologists in classifying BI-RADS 4 mammography lesions. • Trust in the algorithm’s performance was mostly dependent on the certainty of the prediction in combination with a reasonable heatmap. • Personality traits seem to influence human-AI collaboration. Radiologists with specific personality traits were more likely to change their classification according to the algorithm’s prediction than others.Keywords
Funding Information
- Technische Universität München
This publication has 35 references indexed in Scilit:
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Moving beyond local practice: Reconfiguring the adoption of a breast cancer diagnostic technologySocial Science & Medicine, 2015
- Algorithm aversion: People erroneously avoid algorithms after seeing them err.Journal of Experimental Psychology: General, 2015
- The stability of big-five personality traitsEconomics Letters, 2012
- The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational StudiesAnnals of Internal Medicine, 2007
- The international personality item pool and the future of public-domain personality measuresJournal of Research in Personality, 2006
- Evaluation of computer-aided detection (CAD) prompting techniques for mammographyBritish Journal of Radiology, 2005
- The effect of novel prompts upon radiologists" visual search of mammograms2004
- A Perceptually Based Method for Enhancing Pulmonary Nodule RecognitionInvestigative Radiology, 1993
- The development of markers for the Big-Five factor structure.Psychological Assessment, 1992