ChatGPT Yields a Passing Score on a Pediatric Board Preparatory Exam but Raises Red Flags
Open Access
- 1 January 2024
- journal article
- research article
- Published by SAGE Publications in Global Pediatric Health
- Vol. 11
Abstract
Objectives: We aimed to evaluate the performance of a publicly-available online artificial intelligence program (OpenAI’s ChatGPT-3.5 and -4.0, August 3 versions) on a pediatric board preparatory examination, 2021 and 2022 PREP® Self-Assessment, American Academy of Pediatrics (AAP). Methods: We entered 245 questions and answer choices from the Pediatrics 2021 PREP® Self-Assessment and 247 questions and answer choices from the Pediatrics 2022 PREP® Self-Assessment into OpenAI’s ChatGPT-3.5 and ChatGPT-4.0, August 3 versions, in September 2023. The ChatGPT-3.5 and 4.0 scores were compared with the advertised passing scores (70%+) for the PREP® exams and the average scores (74.09%) and (75.71%) for all 10 715 and 6825 first-time human test takers. Results: For the AAP 2021 and 2022 PREP® Self-Assessments, ChatGPT-3.5 answered 143 of 243 (58.85%) and 137 of 247 (55.46%) questions correctly on a single attempt. ChatGPT-4.0 answered 193 of 243 (79.84%) and 208 of 247 (84.21%) questions correctly. Conclusion: Using a publicly-available online chatbot to answer pediatric board preparatory examination questions yielded a passing score but demonstrated significant limitations in the chatbot’s ability to assess some complex medical situations in children, posing a potential risk to this vulnerable population.Keywords
This publication has 36 references indexed in Scilit:
- Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media ForumJAMA Internal Medicine, 2023
- Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for MedicineNew England Journal of Medicine, 2023
- AI in health and medicineNature Medicine, 2022
- Using deep learning and natural language processing models to detect child physical abuseJournal of Pediatric Surgery, 2021
- The promise of artificial intelligence: a review of the opportunities and challenges of artificial intelligence in healthcareBritish Medical Bulletin, 2021
- Artificial intelligence for good health: a scoping review of the ethics literatureBMC Medical Ethics, 2021
- Deep neural networks are superior to dermatologists in melanoma image classificationEuropean Journal of Cancer, 2019
- Evaluation and accurate diagnoses of pediatric diseases using artificial intelligenceNature Medicine, 2019
- Impact of physician workload on burnout in the emergency departmentPsychology, Health & Medicine, 2018
- Physician burnout: contributors, consequences and solutionsJournal of Internal Medicine, 2018