Benchmarking ChatGPT-4 on ACR Radiation Oncology In-Training (TXIT) Exam and Red Journal Gray Zone Cases: Potentials and Challenges for AI-Assisted Medical Education and Decision Making in Radiation Oncology
Authors:
- Johanna GrigoJGJohanna Grigo
- Florian PutzFlorian Putz
Preprint
Abstract
The potential of large language models in medicine for education and decision making purposes has been demonstrated as they achieve decent scores on medical exams such as the United States Medical Licensing Exam (USMLE) and the MedQA exam. In this work, we evaluate the performance of ChatGPT-4 in the specialized field of radiation oncology using the 38th American College of Radiology (ACR) radiation oncology in-training (TXIT) exam and the 2022 red journal gray zone cases. For the TXIT exam, ChatGPT-3.5 and ChatGPT-4 have achieved the scores of 63.65% and 74.57%, respectively, highlighting the advantage of the latest ChatGPT-4 model. Based on the TXIT exam, ChatGPT-4's strong and weak areas in radiation oncology are identified to some extent. Specifically, ChatGPT-4 demonstrates good knowledge of statistics, CNS & eye, pediatrics, biology, and physics but has limitations in bone & soft tissue and gynecology, as per the ACR knowledge domain. Regarding clinical care paths, ChatGPT-4 performs well in diagnosis, prognosis, and toxicity but lacks proficiency in topics related to brachytherapy and dosimetry, as well as in-depth questions from clinical trials. For the gray zone cases, ChatGPT-4 is able to suggest a personalized treatment approach to each case and achieves comparable votes (28.76% on average) to human experts in general. Most importantly, it provides complementary suggestion to the recommendation from a single expert. Both evaluations have demonstrated the potential of ChatGPT in medical education for the general public and cancer patients, as well as the potential to aid clinical decision-making, while acknowledging its limitations in certain domains.Keywords
All Related Versions
This publication has 30 references indexed in Scilit:
- GPT-4: a new era of artificial intelligence in medicineIrish Journal of Medical Science (1971 -), 2023
- Deep learning for brain metastasis detection and segmentation in longitudinal MRI dataMedical Physics, 2022
- A Viewpoint on Isolated Contralateral Axillary Lymph Node Involvement by Breast Cancer: Regional Recurrence or Distant Metastasis?International Journal of Radiation Oncology - Biology - Physics (IJROBP), 2022
- Digital transfer in radiation oncology education for medical students—single-center data and systemic review of the literatureStrahlentherapie und Onkologie, 2022
- Exploring the Role of Resection Post-Radiation Therapy in GliomasInternational Journal of Radiation Oncology - Biology - Physics (IJROBP), 2022
- Radiation Therapy for Cure or Palliation: Case of the Immunosuppressed Patient With Multiple Primary Cancers and Liver TransplantInternational Journal of Radiation Oncology - Biology - Physics (IJROBP), 2022
- Synopsis of Supraclavicular Sarcoma: Synthesis of Stratagem and SolutionsInternational Journal of Radiation Oncology - Biology - Physics (IJROBP), 2022
- A generative adversarial network‐based (GAN‐based) architecture for automatic fiducial marker detection in prostate MRI‐only radiotherapy simulation imagesMedical Physics, 2020
- Deep learning in medical imaging and radiation therapyMedical Physics, 2018
- Introducing: The Red Journal Gray ZoneInternational Journal of Radiation Oncology - Biology - Physics (IJROBP), 2017