Performance of three artificial intelligence (AI)‐based large language models in standardized testing; implications for AI‐assisted dental education
Top Cited Papers
Open Access
- 18 July 2024
- journal article
- research article
- Published by Wiley in Journal of Periodontal Research
- Vol. 60 (2) , 121-133
Abstract
Introduction: The emerging rise in novel computer technologies and automated data analytics has the potential to change the course of dental education. In line with our long‐term goal of harnessing the power of AI to augment didactic teaching, the objective of this study was to quantify and compare the accuracy of responses provided by ChatGPT (GPT‐4 and GPT‐3.5) and Google Gemini, the three primary large language models (LLMs), to human graduate students (control group) to the annual in‐service examination questions posed by the American Academy of Periodontology (AAP).Methods: Under a comparative cross‐sectional study design, a corpus of 1312 questions from the annual in‐service examination of AAP administered between 2020 and 2023 were presented to the LLMs. Their responses were analyzed using chi‐square tests, and the performance was juxtaposed to the scores of periodontal residents from corresponding years, as the human control group. Additionally, two sub‐analyses were performed: one on the performance of the LLMs on each section of the exam; and in answering the most difficult questions.Results: ChatGPT‐4 (total average: 79.57%) outperformed all human control groups as well as GPT‐3.5 and Google Gemini in all exam years (p < .001). This chatbot showed an accuracy range between 78.80% and 80.98% across the various exam years. Gemini consistently recorded superior performance with scores of 70.65% (p = .01), 73.29% (p = .02), 75.73% (p < .01), and 72.18% (p = .0008) for the exams from 2020 to 2023 compared to ChatGPT‐3.5, which achieved 62.5%, 68.24%, 69.83%, and 59.27% respectively. Google Gemini (72.86%) surpassed the average scores achieved by first‐ (63.48% ± 31.67) and second‐year residents (66.25% ± 31.61) when all exam years combined. However, it could not surpass that of third‐year residents (69.06% ± 30.45).Conclusions: Within the confines of this analysis, ChatGPT‐4 exhibited a robust capability in answering AAP in‐service exam questions in terms of accuracy and reliability while Gemini and ChatGPT‐3.5 showed a weaker performance. These findings underscore the potential of deploying LLMs as an educational tool in periodontics and oral implantology domains. However, the current limitations of these models such as inability to effectively process image‐based inquiries, the propensity for generating inconsistent responses to the same prompts, and achieving high (80% by GPT‐4) but not absolute accuracy rates should be considered. An objective comparison of their capability versus their capacity is required to further develop this field of study.Keywords
This publication has 28 references indexed in Scilit:
- Effectiveness of AI-powered Chatbots in responding to orthopaedic postgraduate exam questions—an observational studyInternational Orthopaedics, 2024
- ChatGPT performance in prosthodontics: Assessment of accuracy and repeatability in answer generationThe Journal of Prosthetic Dentistry, 2024
- Artificial intelligence in dental education: ChatGPT's performance on the periodontic in‐service examinationJournal of Periodontology, 2024
- The performance of artificial intelligence language models in board-style dental knowledge assessmentThe Journal of the American Dental Association, 2023
- Creation and Adoption of Large Language Models in MedicineJAMA, 2023
- Large Language Models Answer Medical Questions Accurately, but Can’t Match Clinicians’ KnowledgeJAMA, 2023
- ChatGPT, Bard, and Large Language Models for Biomedical Research: Opportunities and PitfallsAnnals of Biomedical Engineering, 2023
- ChatGPT for healthcare providers and patients: Practical implications within dermatologyJournal of the American Academy of Dermatology, 2023
- The rise of ChatGPT: Exploring its potential in medical educationAnatomical Sciences Education, 2023
- Opinion Paper: “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policyInternational Journal of Information Management, 2023