Validity and Reliability of Responses to Periodontology Questions by 4 Different Artificial Intelligence Chatbots as Public Information Sources
- 30 September 2025
- journal article
- Published by Cumhuriyet University in Cumhuriyet Dental Journal
- Vol. 28 (3) , 390-396
Abstract
Objectives: To assess and check the validity and reliability of the answers given by ChatGpt-4o mini, Deepseek, Copilot and Gemini 1.5 flash daily chatbots to often seeked queries in the area of periodontology. Materials and Methods: Questions were selected from the most frequently asked patient questions by a periodontologist. Each question was asked to the chatbots three times. The answers (n=240) were independently evaluated by two periodontologists on a Likert scale (5=violently agree; 4=agree; 3: neutral; 2=disagree; 1=violently disagree). Disputes in scoring were removed through evidence-based negotiations. In evaluating the validity of the answers: Low threshold was determined as a score ≥4 for whole three answers; high threshold was determined as a score 5 for whole three answers. Fisher's exact test was performed to compare the validity of the answers among the chatbots. Cronbach's alpha was computed to evaluate the consistency and reliability of recurrent answers for each chatbot. Results: All four chatbots answered the questions. In the low-threshold validity test, ChatGpt had 100%, Deepseek and Copilot had 95%, Gemini had 65%. Gemini was significantly different from the others (p0.05), both were significantly higher than Copilot and Gemini (pKeywords
This publication has 20 references indexed in Scilit:
- DeepSeek versus ChatGPT: Multimodal artificial intelligence revolutionizing scientific discovery. From language editing to autonomous content generation—Redefining innovation in research and practiceKnee Surgery, Sports Traumatology, Arthroscopy, 2025
- Evaluation of validity and reliability of AI Chatbots as public sources of information on dental traumaDental Traumatology, 2024
- Assessment of readability, reliability, and quality of ChatGPT®, BARD®, Gemini®, Copilot®, Perplexity® responses on palliative careMedicine, 2024
- Artificial Intelligence Chatbots in Healthcare: Navigating Accuracy, Privacy, and Global ApplicabilityClinical Gastroenterology and Hepatology, 2024
- Utilizing Large Language Models to Simplify Radiology Reports: a comparative analysis of ChatGPT3.5, ChatGPT4.0, Google Bard, and Microsoft BingmedRxiv, 2023
- Evaluating Artificial Intelligence Responses to Public Health QuestionsJAMA Network Open, 2023
- Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media ForumJAMA Internal Medicine, 2023
- An Extensive Comparative Analysis of Chatbot Technologies - ChatGPT, Google BARD and Microsoft Bing2023
- Deep learningNature, 2015
- Deep learning in neural networks: An overviewNeural Networks, 2015