Chatgpt Vs. Google Gemini: Assessment of Performance Regarding the Accuracy and Repeatability of Responses to Questions in Implant-Supported Prostheses
- 31 August 2025
- journal article
- Published by Ankara University in European Annals of Dental Sciences
- Vol. 52 (2) , 71-78
Abstract
Purpose: This study aimed to determine the accuracy and repeatability of the responses of different large language models to questions regarding implant-supported prostheses and assess the impact of pre-prompt utilization and the time of day. Materials & Methods: A total of 12 open-ended questions related to implant-supported prostheses were generated and the content validity of the questions was verified by a specialist. Following that, questions were posed to 2 different LLMs: ChatGPT-4.0 and Google Gemini (morning, afternoon, evening; with and without pre-prompt). The responses were evaluated by two expert prosthodontists with a holistic rubric; the concordance between the graders' responses and repeated responses by C and G software programs was calculated with the Brennan and Prediger coefficient, Cohen kappa coefficient, Fleiss kappa, and Krippendorff alpha coefficients. Kruskal-Wallis, Mann-Whitney U, independent t-test, and ANOVA analyses were used to compare the responses obtained in the implementations. Results: The results showed that the accuracy of ChatGPT and Google Gemini was 34.7% and 17.4%, respectively. The implementation of pre-prompt significantly increased accuracy in Gemini (p = 0.026). No significant difference was found according to the time of day (morning, afternoon, evening) or inter-week implementations. In addition, inter-rater reliability and repeatability showed high levels of consistency. Conclusion: The use of pre-prompt positively affected accuracy and repeatability in both ChatGPT and Google Gemini. However, LLMs can still produce hallucinations. Therefore, LLMs may assist clinicians but they should be aware of these limitations. Keywords: Chatbot, ChatGPT, Prostheses and Implant.Keywords
Funding Information
- None
This publication has 29 references indexed in Scilit:
- Assessing GPT‐4's accuracy in answering clinical pharmacological questions on pain therapyBritish Journal of Clinical Pharmacology, 2025
- Performance of the ChatGPT-3.5, ChatGPT-4, and Google Gemini large language models in responding to dental implantology inquiriesThe Journal of Prosthetic Dentistry, 2025
- Large language models in periodontology: Assessing their performance in clinically relevant questionsThe Journal of Prosthetic Dentistry, 2024
- Accuracy of Large Language Models for Infective Endocarditis Prophylaxis in Dental ProceduresInternational Dental Journal, 2024
- Accuracy and consistency of chatbots versus clinicians for answering pediatric dentistry questions: A pilot studyJournal of Dentistry, 2024
- Evaluation of ChatGPT-generated medical responses: A systematic review and meta-analysisJournal of Biomedical Informatics, 2024
- ChatGPT performance in prosthodontics: Assessment of accuracy and repeatability in answer generationThe Journal of Prosthetic Dentistry, 2024
- Beyond the Scalpel: Assessing ChatGPT's potential as an auxiliary intelligent virtual assistant in oral surgeryComputational and Structural Biotechnology Journal, 2023
- Drawbacks of Artificial Intelligence and Their Potential Solutions in the Healthcare SectorBiomedical Materials & Devices, 2023
- Artificial intelligence models for tooth-supported fixed and removable prosthodontics: A systematic reviewThe Journal of Prosthetic Dentistry, 2023