Assessing the validity of ChatGPT-4o and Google Gemini Advanced when responding to frequently asked questions in endodontics
Open Access
- 1 January 2025
- journal article
- research article
- Published by FapUNIFESP (SciELO) in Journal of Applied Oral Science
- Vol. 33, e20250321
Abstract
Artificial intelligence (AI) is transforming access to dental information via large language models (LLMs) such as ChatGPT and Google Gemini. Both models are increasingly being used in endodontics as a source of information for patients. Therefore, as developers release new versions, the validity of their responses must be continuously compared to professional consultations. Objective: This study aimed to evaluate the validity of the responses provided by the most advanced LLMs [Google Gemini Advanced (GGA) and ChatGPT-4o] to frequently asked questions (FAQs) in endodontics. Methodology: A cross-sectional analytical study was conducted in five phases. The top 20 endodontic FAQs submitted by users to chatbots and collected from Google Trends were compiled. In total, nine academically certified endodontic specialists with educational roles scored GGA and ChatGPT-4o responses to the FAQs using a five-point Likert scale. Validity was determined using high (4.5-5) and low (≥4) thresholds. The Fisher's exact test was used for comparative analysis. Results: At the low threshold, both models obtained 95% validity (95% CI: 75.1%- 99.9%; p=.05). At the high threshold, ChatGPT-4o achieved 35% (95% CI: 15.4%- 59.2%) and GGA, 40% (95% CI: 19.1%- 63.9%) validity (p=1). Conclusions: ChatGPT-4o and GGA responses showed high validity under lenient criteria that significantly decreased under stricter thresholds, limiting their reliability as a stand-alone source of information in endodontics. While AI chatbots show promise to improve patient education in endodontics, their validity limitations under rigorous evaluation highlight the need for careful professional monitoring.Keywords
This publication has 23 references indexed in Scilit:
- Accuracy and Consistency of Gemini Responses Regarding the Management of Traumatized Permanent TeethDental Traumatology, 2024
- Artificial intelligence in endodontics: Fundamental principles, workflow, and tasksInternational Endodontic Journal, 2024
- Innovating dental diagnostics: ChatGPT's accuracy on diagnostic challengesOral Diseases, 2024
- Generative artificial intelligence in healthcare: A scoping review on benefits, challenges and applicationsInternational Journal of Medical Informatics, 2024
- Artificial Intelligence in Endodontic EducationJournal of Endodontics, 2024
- Evaluating GPT-4V’s performance in the Japanese national dental examination: A challenge exploredJournal of Dental Sciences, 2023
- Validity and reliability of artificial intelligence chatbots as public sources of information on endodonticsInternational Endodontic Journal, 2023
- Unveiling the ChatGPT phenomenon: Evaluating the consistency and accuracy of endodontic question answersInternational Endodontic Journal, 2023
- Artificial Intelligence in Endodontics: Current Applications and Future DirectionsJournal of Endodontics, 2021
- High-performance medicine: the convergence of human and artificial intelligenceNature Medicine, 2019