Performance of ChatGPT on questions from the Brazilian College of Radiology annual resident evaluation test
Open Access
- 1 January 2024
- journal article
- research article
- Published by FapUNIFESP (SciELO) in Radiologia Brasileira
- Vol. 57, e20230083
Abstract
Objective: To test the performance of ChatGPT on radiology questions formulated by the Colégio Brasileiro de Radiologia (CBR, Brazilian College of Radiology), evaluating its failures and successes. Materials and Methods: 165 questions from the CBR annual resident assessment (2018, 2019, and 2022) were presented to ChatGPT. For statistical analysis, the questions were divided by the type of cognitive skills assessed (lower or higher order), by topic (physics or clinical), by subspecialty, by style (description of a clinical finding or sign, clinical management of a case, application of a concept, calculation/classification of findings, correlations between diseases, or anatomy), and by target academic year (all, second/third year, or third year only). Results: ChatGPT answered 88 (53.3%) of the questions correctly. It performed significantly better on the questions assessing lower-order cognitive skills than on those assessing higher-order cognitive skills, providing the correct answer on 38 (64.4%) of 59 questions and on only 50 (47.2%) of 106 questions, respectively (p = 0.01). The accuracy rate was significantly higher for physics questions than for clinical questions, correct answers being provided for 18 (90.0%) of 20 physics questions and for 70 (48.3%) of 145 clinical questions (p = 0.02). There was no significant difference in performance among the subspecialties or among the academic years (p > 0.05). Conclusion: Even without dedicated training in this field, ChatGPT demonstrates reasonable performance, albeit still insufficient for approval, on radiology questions formulated by the CBR.Keywords
This publication has 12 references indexed in Scilit:
- ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reportsEuropean Radiology, 2023
- Comparative Performance of ChatGPT and Bard in a Text-Based Radiology Knowledge AssessmentCanadian Association of Radiologists Journal, 2023
- ChatGPT failed Taiwan’s Family Medicine Board ExamJournal of the Chinese Medical Association, 2023
- Performance of ChatGPT on a Radiology Board-style Examination: Insights into Current Strengths and LimitationsRadiology, 2023
- ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerationsFrontiers in Artificial Intelligence, 2023
- ChatGPT and the Future of Medical WritingRadiology, 2023
- Aye, AI! ChatGPT passes multiple-choice family medicine examMedical Teacher, 2023
- Artificial Hallucinations in ChatGPT: Implications in Scientific WritingCureus, 2023
- How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge AssessmentJMIR Medical Education, 2023
- AI in Health: State of the Art, Challenges, and Future DirectionsYearbook of Medical Informatics, 2019