Evaluating large language models on a highly-specialized topic, radiation oncology physics
Top Cited Papers
Open Access
- 17 July 2023
- journal article
- research article
- Published by Frontiers Media SA in Frontiers in Oncology
- Vol. 13, 1219326
Abstract
Purpose: We present the first study to investigate Large Language Models (LLMs) in answering radiation oncology physics questions. Because popular exams like AP Physics, LSAT, and GRE have large test-taker populations and ample test preparation resources in circulation, they may not allow for accurately assessing the true potential of LLMs. This paper proposes evaluating LLMs on a highly-specialized topic, radiation oncology physics, which may be more pertinent to scientific and medical communities in addition to being a valuable benchmark of LLMs.Methods: We developed an exam consisting of 100 radiation oncology physics questions based on our expertise. Four LLMs, ChatGPT (GPT-3.5), ChatGPT (GPT-4), Bard (LaMDA), and BLOOMZ, were evaluated against medical physicists and non-experts. The performance of ChatGPT (GPT-4) was further explored by being asked to explain first, then answer. The deductive reasoning capability of ChatGPT (GPT-4) was evaluated using a novel approach (substituting the correct answer with “None of the above choices is the correct answer.”). A majority vote analysis was used to approximate how well each group could score when working together.Results: ChatGPT GPT-4 outperformed all other LLMs and medical physicists, on average, with improved accuracy when prompted to explain before answering. ChatGPT (GPT-3.5 and GPT-4) showed a high level of consistency in its answer choices across a number of trials, whether correct or incorrect, a characteristic that was not observed in the human test groups or Bard (LaMDA). In evaluating deductive reasoning ability, ChatGPT (GPT-4) demonstrated surprising accuracy, suggesting the potential presence of an emergent ability. Finally, although ChatGPT (GPT-4) performed well overall, its intrinsic properties did not allow for further improvement when scoring based on a majority vote across trials. In contrast, a team of medical physicists were able to greatly outperform ChatGPT (GPT-4) using a majority vote.Conclusion: This study suggests a great potential for LLMs to work alongside radiation oncology experts as highly knowledgeable assistants.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language ProcessingACM Computing Surveys, 2023
- AgriBERT: Knowledge-Infused Agricultural Language Models for Matching Food and Nutrition2022
- AMMU: A survey of transformer-based biomedical pretrained language modelsJournal of Biomedical Informatics, 2022
- ChestXRayBERT: A Pretrained Language Model for Chest Radiology Report SummarizationIEEE Transactions on Multimedia, 2021
- Making Pre-trained Language Models Better Few-shot Learners2021
- Technical Note: Integrating an open source Monte Carlo code “MCsquare” for clinical use in intensity‐modulated proton therapyMedical Physics, 2020
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- Robust optimization in IMPT using quadratic objective functions to account for the minimum MU constraintMedical Physics, 2017
- Proton beam therapy for locally advanced lung cancer: A reviewWorld Journal of Clinical Oncology, 2014
- PTV-based IMPT optimization incorporating planning risk volumes vs robust optimizationMedical Physics, 2013