Fennie van der Graaf
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Fennie van der Graaf has a total of 19 co-authors. Here are the top co-authors by number of shared publications:
- Alessa Hering
Radboudumc
3 shared publications - Colin Jacobs
From the Department of Medical Imaging, Radboud Institute of Medical Innovation, Radboud University Medical Center, Nijmegen, the Netherlands
3 shared publications - Lena Philipp
Department of Medical Imaging, Radboudumc, Nijmegen, The Netherlands
3 shared publications - Ewoud J. Smit
Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, The Netherlands
2 shared publications - Jan Tagscherer
Department of Medical Imaging, Radboudumc, Nijmegen, The Netherlands
2 shared publications - Sarah de Boer
Department of Medical Imaging, Radboudumc, Nijmegen, The Netherlands
2 shared publications
Latest Publications
- Open Access
Large Language Models to Label CT Radiology Reports for Incidental Findings Found in Lung Cancer Screening: A Comparative Analysis
Objective: To compare LLM and prompting strategies for labeling Specific Incidental Findings (SIFs) occurring during lung cancer screening, in radiology reports. Materials and Methods: In this retrospective study, 300 CT reports from a Dutch academic hospital were included between 19 th March 2003 and 28 th December 2020. GPT-5, GPT-4-Turbo, Llama-3.1-8B-INST, MedGemma-4B-IT, a keyword-based search method, and a radiologist labelled the reports for the presence or absence of nine SIFs. The radiologist’s scores were the reference standard. Each LLM was evaluated with four different prompting strategies: two without Chain-of-Thought (CoT), and two with CoT (one zero shot, one few shot). The automated labelling methods were evaluated against the radiologist primarily using Cohen’s Kappa, and additionally sensitivity, specificity, and F1 scores. Results: GPT-5 achieved substantial or almost perfect agreement for seven out of nine SIFs: aortic valve disease, bronchiectasis, consolidation, coronary artery calcification, emphysema, abnormality in mediastinal lymph nodes, and thyroid abnormalities, with Cohen κ values of 0.81 (0.70, 0.93), 0.97 (0.93, 1.00), 0.71 (0.62, 0.82), 0.92 (0.87, 0.96), 0.91 (0.84, 0.97), 0.71 (0.62, 0.80), and 0.92 (0.86, 0.98) respectively. For the remaining two SIFs, moderate LLM agreement was obtained with MedGemma-4B-IT and with GPT-4-Turbo with Cohen κ values of 0.48 (0.40, 0.60), 0.42 (0.31, 0.58) for interstitial lung abnormalities and mediastinal mass respectively. The highest agreement was obtained with the non-CoT prompt strategies for most SIFs. Conclusion: This study demonstrated that LLMs can automatically label radiology reports with agreement ranging from almost perfect to moderate, but not one LLM – prompt combination achieved the highest agreement across all SIFs.
Open AccessLarge Language Models to Label CT Radiology Reports for Incidental Findings Found in Lung Cancer Screening: A Comparative Analysis
Objective: To compare LLM and prompting strategies for labeling Specific Incidental Findings (SIFs) occurring during lung cancer screening, in radiology reports. Materials and Methods: In this retrospective study, 300 CT reports from a Dutch academic hospital were included between 19 th March 2003 and 28 th December 2020. GPT-5, GPT-4-Turbo, Llama-3.1-8B-INST, MedGemma-4B-IT, a keyword-based search method, and a radiologist labelled the reports for the presence or absence of nine SIFs. The radiologist’s scores were the reference standard. Each LLM was evaluated with four different prompting strategies: two without Chain-of-Thought (CoT), and two with CoT (one zero shot, one few shot). The automated labelling methods were evaluated against the radiologist primarily using Cohen’s Kappa, and additionally sensitivity, specificity, and F1 scores. Results: GPT-5 achieved substantial or almost perfect agreement for seven out of nine SIFs: aortic valve disease, bronchiectasis, consolidation, coronary artery calcification, emphysema, abnormality in mediastinal lymph nodes, and thyroid abnormalities, with Cohen κ values of 0.81 (0.70, 0.93), 0.97 (0.93, 1.00), 0.71 (0.62, 0.82), 0.92 (0.87, 0.96), 0.91 (0.84, 0.97), 0.71 (0.62, 0.80), and 0.92 (0.86, 0.98) respectively. For the remaining two SIFs, moderate LLM agreement was obtained with MedGemma-4B-IT and with GPT-4-Turbo with Cohen κ values of 0.48 (0.40, 0.60), 0.42 (0.31, 0.58) for interstitial lung abnormalities and mediastinal mass respectively. The highest agreement was obtained with the non-CoT prompt strategies for most SIFs. Conclusion: This study demonstrated that LLMs can automatically label radiology reports with agreement ranging from almost perfect to moderate, but not one LLM – prompt combination achieved the highest agreement across all SIFs.
- Open Access
- 27 August 2026
- Published by Springer Nature in International Journal of Computer Assisted Radiology and Surgery
- p. 1-12
Systematic evaluation of foundation models for organ-level classification on CT scans
Purpose: Foundation models are increasingly used as frozen feature extractors for CT classification tasks, yet the determinants of their downstream performance remain unclear. We assess how foundation model choice, feature aggregation strategy, and abnormality type are associated with performance in organ-level abnormality classification. We further evaluate whether more expressive aggregation strategies outperform simple pooling, and whether performance varies by abnormality type. Methods: We evaluate five state-of-the-art medical imaging foundation models on binary organ-level abnormality classification (normal vs. abnormal) across six abdominal organs. Local patch-level embeddings are aggregated using six strategies, including simple pooling methods (e.g., mean pooling) and attention-based multiple instance learning. A linear classifier is trained on feature embeddings, and performance is evaluated on an annotated test set of 200 CT scans. Results: Among the evaluated models, 3D CT-native models generally outperformed 2D multi-modal models. None of the tested aggregation strategies significantly outperformed mean pooling, including attention-based multiple instance learning (best-performing aggregation vs. mean: $$\Delta $$ AUC = 0.008, 95% CI $$[-0.002, 0.022]$$ ). In our organ-level classification setting, performance differed between abnormality type for all well-performing models (SPECTRE, TAP-CT, CT-FM), with lower AUCs observed for focal abnormalities compared to diffuse abnormalities (largest difference: $$\Delta $$ AUC = 0.108, 95% CI [0.080, 0.135]). This gap was not reduced by the evaluated aggregation strategies. Conclusion: In our experiments, downstream performance varied more across foundation models than across aggregation strategies. The choice of aggregation strategy, including attention-based multiple instance learning, did not significantly impact performance in this setting. The persistent gap for focal abnormalities suggests that current representations may insufficiently encode localized disease patterns, motivating the development of localization-aware pre-training approaches.- 27 August 2026
- Published by Springer Nature in International Journal of Computer Assisted Radiology and Surgery
- p. 1-12
Open AccessSystematic evaluation of foundation models for organ-level classification on CT scans
Purpose: Foundation models are increasingly used as frozen feature extractors for CT classification tasks, yet the determinants of their downstream performance remain unclear. We assess how foundation model choice, feature aggregation strategy, and abnormality type are associated with performance in organ-level abnormality classification. We further evaluate whether more expressive aggregation strategies outperform simple pooling, and whether performance varies by abnormality type. Methods: We evaluate five state-of-the-art medical imaging foundation models on binary organ-level abnormality classification (normal vs. abnormal) across six abdominal organs. Local patch-level embeddings are aggregated using six strategies, including simple pooling methods (e.g., mean pooling) and attention-based multiple instance learning. A linear classifier is trained on feature embeddings, and performance is evaluated on an annotated test set of 200 CT scans. Results: Among the evaluated models, 3D CT-native models generally outperformed 2D multi-modal models. None of the tested aggregation strategies significantly outperformed mean pooling, including attention-based multiple instance learning (best-performing aggregation vs. mean: $$\Delta $$ AUC = 0.008, 95% CI $$[-0.002, 0.022]$$ ). In our organ-level classification setting, performance differed between abnormality type for all well-performing models (SPECTRE, TAP-CT, CT-FM), with lower AUCs observed for focal abnormalities compared to diffuse abnormalities (largest difference: $$\Delta $$ AUC = 0.108, 95% CI [0.080, 0.135]). This gap was not reduced by the evaluated aggregation strategies. Conclusion: In our experiments, downstream performance varied more across foundation models than across aggregation strategies. The choice of aggregation strategy, including attention-based multiple instance learning, did not significantly impact performance in this setting. The persistent gap for focal abnormalities suggests that current representations may insufficiently encode localized disease patterns, motivating the development of localization-aware pre-training approaches. - 6 April 2026
- Published by SAGE Publications in Biopreservation and Biobanking
- Vol. 24 (3) , 241-246
A Case Study: Testing of an Artificial Intelligence Chatbot Designed for Guiding and Assisting Participants at a Biobank Research Site
This study investigated if improving a chatbot using user test feedback can enhance the perceived value of a chatbot for a specific biobanking use case. The chatbot was designed to guide participants in a research study around a biobank site to facilitate their data collection experience by conversing with them about what to do and where to go and answering any questions about the measurements, the research, or the biobank. A total of 32 test participants tested the chatbot and provided feedback about their experience, and the answers were analyzed to evaluate perceived value. The first group of users tested the initial version of the chatbot, and the second group tested the improved version of the chatbot. The results demonstrated that by including user feedback to design the chatbot, the perceived value in the second testing increased. Following these early-stage study results, it was determined that the chatbot was ready for deployment in the biobank.
- 6 April 2026
- Published by SAGE Publications in Biopreservation and Biobanking
- Vol. 24 (3) , 241-246
A Case Study: Testing of an Artificial Intelligence Chatbot Designed for Guiding and Assisting Participants at a Biobank Research Site
This study investigated if improving a chatbot using user test feedback can enhance the perceived value of a chatbot for a specific biobanking use case. The chatbot was designed to guide participants in a research study around a biobank site to facilitate their data collection experience by conversing with them about what to do and where to go and answering any questions about the measurements, the research, or the biobank. A total of 32 test participants tested the chatbot and provided feedback about their experience, and the answers were analyzed to evaluate perceived value. The first group of users tested the initial version of the chatbot, and the second group tested the improved version of the chatbot. The results demonstrated that by including user feedback to design the chatbot, the perceived value in the second testing increased. Following these early-stage study results, it was determined that the chatbot was ready for deployment in the biobank.
- 12 March 2026
- Published by Springer Nature
- p. 48-54
Modular Pipeline for Rapidly Evaluating Foundation Models in Medical Imaging
- 12 March 2026
- Published by Springer Nature
- p. 48-54
Modular Pipeline for Rapidly Evaluating Foundation Models in Medical Imaging
- Open Access
- 21 August 2025
- Published by Machine Learning for Biomedical Imaging in Machine Learning for Biomedical Imaging
- Vol. 3 (Special on) , 559-593
Fairness Evaluation of Risk Estimation Models for Lung Cancer Screening
Lung cancer is the leading cause of cancer-related mortality in adults worldwide. Screening high-risk individuals with annual low-dose CT (LDCT) can support earlier detection and reduce deaths, but widespread implementation may strain the already limited radiology workforce. Artificial intelligence (AI) models have shown potential in estimating lung cancer risk from LDCT scans. However, high-risk populations for lung cancer are diverse, and these models’ performance across diverse demographic groups remains an open question. In this study, we used the JustEFAB ethical framework to evaluate potential performance disparities and fairness in two AI-based risk estimation models for lung cancer screening: the Sybil lung cancer risk model and the Venkadesh21 nodule risk estimator. We also examined disparities in the PanCan2b logistic regression model recommended in the British Thoracic Society nodule management guideline. Both AI-based models were trained on data from the U.S.-based National Lung Screening Trial (NLST), and assessed on a held-out NLST validation set. We evaluated area under the ROC curve (AUROC), sensitivity, and specificity across demographic subgroups, and explored potential confounding from clinical risk factors. We observed a statistically significant AUROC difference in Sybil’s performance between women (0.88, 95% CI: 0.86, 0.90) and men (0.81, 95% CI: 0.78, 0.84, p < .001). At 90% specificity, Venkadesh21 showed lower sensitivity for Black (0.39, 95% CI: 0.23, 0.59) than White participants (0.69, 95% CI: 0.65, 0.73). These differences were not explained by available clinical confounders and may be classified as unfair biases according to JustEFAB. Our findings highlight the importance of improving and monitoring model performance across underrepresented subgroups in lung cancer screening, as well as further research on algorithmic fairness in this field.
- 21 August 2025
- Published by Machine Learning for Biomedical Imaging in Machine Learning for Biomedical Imaging
- Vol. 3 (Special on) , 559-593
Open AccessFairness Evaluation of Risk Estimation Models for Lung Cancer Screening
Lung cancer is the leading cause of cancer-related mortality in adults worldwide. Screening high-risk individuals with annual low-dose CT (LDCT) can support earlier detection and reduce deaths, but widespread implementation may strain the already limited radiology workforce. Artificial intelligence (AI) models have shown potential in estimating lung cancer risk from LDCT scans. However, high-risk populations for lung cancer are diverse, and these models’ performance across diverse demographic groups remains an open question. In this study, we used the JustEFAB ethical framework to evaluate potential performance disparities and fairness in two AI-based risk estimation models for lung cancer screening: the Sybil lung cancer risk model and the Venkadesh21 nodule risk estimator. We also examined disparities in the PanCan2b logistic regression model recommended in the British Thoracic Society nodule management guideline. Both AI-based models were trained on data from the U.S.-based National Lung Screening Trial (NLST), and assessed on a held-out NLST validation set. We evaluated area under the ROC curve (AUROC), sensitivity, and specificity across demographic subgroups, and explored potential confounding from clinical risk factors. We observed a statistically significant AUROC difference in Sybil’s performance between women (0.88, 95% CI: 0.86, 0.90) and men (0.81, 95% CI: 0.78, 0.84, p < .001). At 90% specificity, Venkadesh21 showed lower sensitivity for Black (0.39, 95% CI: 0.23, 0.59) than White participants (0.69, 95% CI: 0.65, 0.73). These differences were not explained by available clinical confounders and may be classified as unfair biases according to JustEFAB. Our findings highlight the importance of improving and monitoring model performance across underrepresented subgroups in lung cancer screening, as well as further research on algorithmic fairness in this field.