Evaluating the Current Ability of ChatGPT to Assist in Professional Otolaryngology Education
Authors:
Open Access
Abstract
Objective: To quantify ChatGPT's concordance with expert Otolaryngologists when posed with high‐level questions that require blending rote memorization and critical thinking.Study Design: Cross‐sectional survey.Setting: OpenAI's ChatGPT‐3.5 Platform.Methods: Two board‐certified otolaryngologists (HZ, RS) input 2 sets of 30 text‐based questions (open‐ended and single‐answer multiple‐choice) into the ChatGPT‐3.5 model. Responses were rated on a scale (correct, partially correct, incorrect) by each Otolaryngologist working simultaneously with the AI model. Interrater agreement percentage was based on binomial distribution for calculating the 95% confidence intervals and performing significance tests. Statistical significance was defined as P < .05 for 2‐sided tests.Results: In testing open‐ended questions, the ChatGPT model had 56.7% of initially answering questions with complete accuracy, and 86.7% chance of answer with some accuracy (corrected agreement = 80.1%; P < .001). For repeat questions, ChatGPT improved to 73.3% with complete accuracy and 96.7% with some accuracy (corrected agreement = 88.8%; P < .001). For multiple‐choice questions, the ChatGPT model performed substantially worse (43.3% correct).Conclusion: ChatGPT currently does not provide reliably accurate responses to sophisticated questions in Otolaryngology. Professional societies must be aware of the potential of this tool and prevent unscrupulous use during test‐taking situations and consider guidelines for clinical scenarios. Expert clinical oversight is still necessary for myriad use cases (eg, hallucination).Keywords
This publication has 8 references indexed in Scilit:
- Journal of Educational Evaluation for Health Professions received the Journal Impact Factor, 4.4 for the first time on June 28, 2023Journal of Educational Evaluation for Health Professions, 2023
- ChatGPT’s quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questionsEuropean Archives of Oto-Rhino-Laryngology, 2023
- Potential role of ChatGPT in clinical otolaryngology explained by ChatGPTAmerican Journal of Otolaryngology, 2023
- The exciting potential for ChatGPT in obstetrics and gynecologyAmerican Journal of Obstetrics and Gynecology, 2023
- Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language modelsPLOS Digital Health, 2023
- ChatGPT passing USMLE shines a spotlight on the flaws of medical educationPLOS Digital Health, 2023
- How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge AssessmentJMIR Medical Education, 2023
- The Measurement of Observer Agreement for Categorical DataBiometrics, 1977