Evidence-based potential of generative artificial intelligence large language models in orthodontics: a comparative study of ChatGPT, Google Bard, and Microsoft Bing
Top Cited Papers
Open Access
- 13 April 2024
- journal article
- research article
- Published by Oxford University Press (OUP) in European Journal of Orthodontics
- Vol. 48 (1)
Abstract
Summary: Background: The increasing utilization of large language models (LLMs) in Generative Artificial Intelligence across various medical and dental fields, and specifically orthodontics, raises questions about their accuracy. Objective: This study aimed to assess and compare the answers offered by four LLMs: Google’s Bard, OpenAI’s ChatGPT-3.5, and ChatGPT-4, and Microsoft’s Bing, in response to clinically relevant questions within the field of orthodontics. Materials and methods: Ten open-type clinical orthodontics-related questions were posed to the LLMs. The responses provided by the LLMs were assessed on a scale ranging from 0 (minimum) to 10 (maximum) points, benchmarked against robust scientific evidence, including consensus statements and systematic reviews, using a predefined rubric. After a 4-week interval from the initial evaluation, the answers were reevaluated to gauge intra-evaluator reliability. Statistical comparisons were conducted on the scores using Friedman’s and Wilcoxon’s tests to identify the model providing the answers with the most comprehensiveness, scientific accuracy, clarity, and relevance. Results: Overall, no statistically significant differences between the scores given by the two evaluators, on both scoring occasions, were detected, so an average score for every LLM was computed. The LLM answers scoring the highest, were those of Microsoft Bing Chat (average score = 7.1), followed by ChatGPT 4 (average score = 4.7), Google Bard (average score = 4.6), and finally ChatGPT 3.5 (average score 3.8). While Microsoft Bing Chat statistically outperformed ChatGPT-3.5 (P-value = 0.017) and Google Bard (P-value = 0.029), as well, and Chat GPT-4 outperformed Chat GPT-3.5 (P-value = 0.011), all models occasionally produced answers with a lack of comprehensiveness, scientific accuracy, clarity, and relevance. Limitations: The questions asked were indicative and did not cover the entire field of orthodontics. Conclusions: Language models (LLMs) show great potential in supporting evidence-based orthodontics. However, their current limitations pose a potential risk of making incorrect healthcare decisions if utilized without careful consideration. Consequently, these tools cannot serve as a substitute for the orthodontist’s essential critical thinking and comprehensive subject knowledge. For effective integration into practice, further research, clinical validation, and enhancements to the models are essential. Clinicians must be mindful of the limitations of LLMs, as their imprudent utilization could have adverse effects on patient care.Keywords
This publication has 23 references indexed in Scilit:
- ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid ConcernsHealthcare, 2023
- Strategies for developing evidence-based clinical practice guidelines to foster implementation into dental practiceThe Journal of the American Dental Association, 2022
- Adopting artificial intelligence in dental education: A model for academic leadership and innovationJournal of Dental Education, 2022
- Artificial intelligence for caries detection: Randomized trialJournal of Dentistry, 2021
- Applications of artificial intelligence in dentistry: A comprehensive reviewJournal of Esthetic and Restorative Dentistry, 2021
- Considering the possibilities and pitfalls of Generative Pre-trained Transformer 3 (GPT-3) in healthcare deliverynpj Digital Medicine, 2021
- Application and performance of artificial intelligence technology in forensic odontology – A systematic reviewLegal Medicine, 2021
- Dental Caries Diagnosis and Detection Using Neural Networks: A Systematic ReviewJournal of Clinical Medicine, 2020
- The use and performance of artificial intelligence applications in dental and maxillofacial radiology: A systematic reviewDentomaxillofacial Radiology, 2020
- Evidence-based dentistry: an overview of the challenges in changing professional practiceBritish Dental Journal, 2001