Advancing Medical Education: Performance of Generative Artificial Intelligence Models on Otolaryngology Board Preparation Questions With Image Analysis Insights
Open Access
- 9 July 2024
- journal article
- research article
- Published by Springer Nature in Cureus
- Vol. 16 (7) , e64204
Abstract
Objective To evaluate and compare the performance of Chat Generative Pre-Trained Transformer (ChatGPT), GPT-4, and Google Bard on United States otolaryngology board-style questions to scale their ability to act as an adjunctive study tool and resource for students and doctors. Methods A 1077 text question and 60 image-based questions from the otolaryngology board exam preparation tool BoardVitals were inputted into ChatGPT, GPT-4, and Google Bard. The questions were scaled true or false, depending on whether the artificial intelligence (AI) modality provided the correct response. Data analysis was performed in R Studio. Results GPT-4 scored the highest at 78.7% compared to ChatGPT and Bard at 55.3% and 61.7% (p<0.001), respectively. In terms of question difficulty, all three AI models performed best on easy questions (ChatGPT: 69.7%, GPT-4: 92.5%, and Bard: 76.4%) and worst on hard questions (ChatGPT: 42.3%, GPT-4: 61.3%, and Bard: 45.6%). Across all difficulty levels, GPT-4 did better than Bard and ChatGPT (p<0.0001). GPT-4 outperformed ChatGPT and Bard in all subspecialty sections, with significantly higher scores (p0.05). On image-based questions, GPT-4 performed better than Bard (56.7% vs 46.4%, p=0.368) and had better overall image interpretation capabilities. Conclusion This study showed that the GPT-4 model performed better than both ChatGPT and Bard on the United States otolaryngology board practice questions. Although the GPT-4 results were promising, AI should still be used with caution when being implemented in medical education or patient care settings.Keywords
This publication has 17 references indexed in Scilit:
- Physician views of artificial intelligence in otolaryngology and rhinology: A mixed methods studyLaryngoscope Investigative Otolaryngology, 2023
- Comparative Performance of ChatGPT and Bard in a Text-Based Radiology Knowledge AssessmentCanadian Association of Radiologists Journal, 2023
- Performance of ChatGPT, GPT-4, and Google Bard on a Neurosurgery Oral Boards Preparation Question BankNeurosurgery, 2023
- ChatGPT’s quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questionsEuropean Archives of Oto-Rhino-Laryngology, 2023
- Evaluation of the performance of GPT-3.5 and GPT-4 on the Medical Final ExaminationmedRxiv, 2023
- Evaluating ChatGPT-4 in Otolaryngology–Head and Neck Surgery Board Examination using the CVSA ModelmedRxiv, 2023
- ChatGPT takes on the European Exam in Core Cardiology: an artificial intelligence success story?European Heart Journal - Digital Health, 2023
- Large language models and the perils of their hallucinationsCritical Care, 2023
- Performance of ChatGPT on the MCAT: The Road to Personalized and Equitable Premedical LearningmedRxiv, 2023
- AI-Based Modeling: Techniques, Applications and Research Issues Towards Automation, Intelligent and Smart SystemsSN Computer Science, 2022