Benchmarking ChatGPT-4 on a radiation oncology in-training exam and Red Journal Gray Zone cases: potentials and challenges for ai-assisted medical education and decision making in radiation oncology
Authors:
Top Cited Papers
Open Access
- 14 September 2023
- journal article
- research article
- Published by Frontiers Media SA in Frontiers in Oncology
- Vol. 13, 1265024
Abstract
Purpose: The potential of large language models in medicine for education and decision-making purposes has been demonstrated as they have achieved decent scores on medical exams such as the United States Medical Licensing Exam (USMLE) and the MedQA exam. This work aims to evaluate the performance of ChatGPT-4 in the specialized field of radiation oncology.Methods: The 38th American College of Radiology (ACR) radiation oncology in-training (TXIT) exam and the 2022 Red Journal Gray Zone cases are used to benchmark the performance of ChatGPT-4. The TXIT exam contains 300 questions covering various topics of radiation oncology. The 2022 Gray Zone collection contains 15 complex clinical cases.Results: For the TXIT exam, ChatGPT-3.5 and ChatGPT-4 have achieved the scores of 62.05% and 78.77%, respectively, highlighting the advantage of the latest ChatGPT-4 model. Based on the TXIT exam, ChatGPT-4’s strong and weak areas in radiation oncology are identified to some extent. Specifically, ChatGPT-4 demonstrates better knowledge of statistics, CNS & eye, pediatrics, biology, and physics than knowledge of bone & soft tissue and gynecology, as per the ACR knowledge domain. Regarding clinical care paths, ChatGPT-4 performs better in diagnosis, prognosis, and toxicity than brachytherapy and dosimetry. It lacks proficiency in in-depth details of clinical trials. For the Gray Zone cases, ChatGPT-4 is able to suggest a personalized treatment approach to each case with high correctness and comprehensiveness. Importantly, it provides novel treatment aspects for many cases, which are not suggested by any human experts.Conclusion: Both evaluations demonstrate the potential of ChatGPT-4 in medical education for the general public and cancer patients, as well as the potential to aid clinical decision-making, while acknowledging its limitations in certain domains. Owing to the risk of hallucinations, it is essential to verify the content generated by models such as ChatGPT for accuracy.Keywords
All Related Versions
This publication has 38 references indexed in Scilit:
- NRG Oncology/NSABP B-51/RTOG 1304: Phase III trial to determine if chest wall and regional nodal radiotherapy (CWRNRT) post mastectomy (Mx) or the addition of RNRT to whole breast RT post breast-conserving surgery (BCS) reduces invasive breast cancer recurrence-free interval (IBCR-FI) in patients (pts) with pathologically positive axillary (PPAx) nodes who are ypN0 after neoadjuvant chemotherapy (NC).Journal of Clinical Oncology, 2019
- Deep learning in medical imaging and radiation therapyMedical Physics, 2018
- Chemotherapy versus chemoradiotherapy after surgery and preoperative chemotherapy for resectable gastric cancer (CRITICS): an international, open-label, randomised phase 3 trialThe Lancet Oncology, 2018
- Endocrine Therapy for Early Breast Cancer: Updated Review2017
- Optimal fractionation of preoperative radiotherapy and timing to surgery for rectal cancer (Stockholm III): a multicentre, randomised, non-blinded, phase 3, non-inferiority trialThe Lancet Oncology, 2017
- Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentationMedical Image Analysis, 2017
- The Eighth Edition AJCC Cancer Staging Manual: Continuing to build a bridge from a population‐based to a more “personalized” approach to cancer stagingCA: A Cancer Journal for Clinicians, 2017
- Introducing: The Red Journal Gray ZoneInternational Journal of Radiation Oncology - Biology - Physics (IJROBP), 2017
- Tumor treating fields: a novel treatment modality and its use in brain tumorsNeuro-Oncology, 2016
- Dose–effect relationship and risk factors for vaginal stenosis after definitive radio(chemo)therapy with image-guided brachytherapy for locally advanced cervical cancer in the EMBRACE studyRadiotherapy and Oncology, 2016