Chat Generative Pretrained Transformer Fails the Multiple-Choice American College of Gastroenterology Self-Assessment Test
Top Cited Papers
- 22 May 2023
- journal article
- research article
- Published by Wolters Kluwer Health in American Journal of Gastroenterology
- Vol. 118 (12) , 2280-2282
Abstract
INTRODUCTION: Chat Generative Pretrained Transformer (ChatGPT) is a natural language processing model that generates human-like text. METHODS: ChatGPT-3 and ChatGPT-4 were used to answer the 2022 and 2021 American College of Gastroenterology self-assessment tests. The exact questions were inputted in both versions of ChatGPT. A score of 70% or higher was required to pass the assessment. RESULTS: Overall, ChatGPT-3 scored 65.1% on 455 included questions and GPT-4 scored 62.4%. DISCUSSION: ChatGPT did not pass the American College of Gastroenterology self-assessment test. We do not recommend its use for medical education in gastroenterology in its current form.Keywords
This publication has 5 references indexed in Scilit:
- Aye, AI! ChatGPT passes multiple-choice family medicine examMedical Teacher, 2023
- Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language modelsPLOS Digital Health, 2023
- How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge AssessmentJMIR Medical Education, 2023
- Introducing Artificial Intelligence Training in Medical EducationJMIR Medical Education, 2019
- Medical Education Must Move From the Information Age to the Age of Artificial IntelligenceAcademic Medicine, 2018