A bilingual evaluation of chatbot performance in bruxism-related information: accuracy and readability across two models
Open Access
- 16 May 2026
- journal article
- Published by Springer Nature in BMC Oral Health
- Vol. 26 (1)
Abstract
Artificial intelligence (AI) chatbots have increasing applications in healthcare; however, their accuracy and readability across different languages remain unclear. Therefore, this study aimed to compare the performance of ChatGPT-5 and DeepSeek-V3 on bruxism-related questions in English and Turkish. Responses generated by ChatGPT-5 and DeepSeek-V3 to 20 questions were evaluated in Turkish and English, yielding four chatbot-language conditions. Accuracy was independently scored by two prosthodontists using a Modified Global Quality Score, and inter-rater reliability was assessed using the intraclass correlation coefficient (ICC). Readability was defined as the ease of reading and was assessed using the Flesch Reading Ease (FRE) scale for English texts and the Ateşman readability formula for Turkish texts. Descriptive statistics were calculated, differences in accuracy across conditions were analyzed using the Friedman test, and the relationship between accuracy and readability was examined using Spearman's rank correlation coefficient. Accuracy scores of responses generated by ChatGPT-5 and DeepSeek-V3 were generally high across both Turkish and English. No statistically significant differences in accuracy were observed among the four model-language combinations based on the Friedman test (χ² (3) = 3.204, p=0.361). Inter-rater reliability analysis demonstrated moderate agreement for single measures and good agreement for average measures between the two evaluators (ICC = 0.623 and 0.767, respectively). Comparison of Atesman readability scores and FRE-based English readability analyses revealed no statistically significant differences between the two chatbots, as confirmed by Mann-Whitney U tests (Turkish: p = 0.646; English: p = 0.745). ChatGPT-5 and DeepSeek-V3 demonstrated comparable accuracy in bruxism-related responses across Turkish and English. However, high readability levels limit patient accessibility, and no association was found between accuracy and readability, indicating that simplification is necessary for effective patient education.Keywords
This publication has 38 references indexed in Scilit:
- Can Chatbots Provide Accurate and Readable Information for Patients With Temporomandibular Disorders?Journal of Oral and Maxillofacial Surgery, 2025
- DeepSeek versus ChatGPT: Multimodal artificial intelligence revolutionizing scientific discovery. From language editing to autonomous content generation—Redefining innovation in research and practiceKnee Surgery, Sports Traumatology, Arthroscopy, 2025
- Accuracy of Spanish and English-generated ChatGPT responses to commonly asked patient questions about labor epidurals: a survey-based study among bilingual obstetric anesthesia expertsInternational Journal of Obstetric Anesthesia, 2024
- Reliability and Readability evaluation of chatbots responses as a patient information resource for the most common PET/CT scansRevista Española de Medicina Nuclear e Imagen Molecular (English Edition), 2024
- Comprehensiveness of Large Language Models in Patient Queries on Gingival and Endodontic HealthInternational Dental Journal, 2024
- Is ChatGPT an Accurate and Readable Patient Aid for Third Molar Extractions?Journal of Oral and Maxillofacial Surgery, 2024
- Evaluation of AI-generated responses by different artificial intelligence chatbots to the clinical decision-making case-based questions in oral and maxillofacial surgeryOral Surgery, Oral Medicine, Oral Pathology and Oral Radiology, 2024
- ChatGPT performance in prosthodontics: Assessment of accuracy and repeatability in answer generationThe Journal of Prosthetic Dentistry, 2024
- Examination of the reliability and readability of Chatbot Generative Pretrained Transformer’s (ChatGPT) responses to questions about orthodontics and the evolution of these responses in an updated versionAmerican Journal of Orthodontics and Dentofacial Orthopedics, 2024
- A new readability yardstick.Journal of Applied Psychology, 1948