Performance of large language models in reporting oral health concerns and side effects in head and neck cancer: a comparative study
Open Access
- 20 December 2025
- journal article
- research article
- Published by Springer Nature in Journal of Cancer Research and Clinical Oncology
- Vol. 152 (1) , 17
Abstract
Purpose: With increasing reliance on large language models (LLMs) for health information, this study evaluated reliability and quality, understandability, actionability, readability and misinformation risk of responses from LLMs to oral health concerns and oral side effects in head and neck cancer (HNC) patients. Methods: Frequently asked questions on oral health and HNC therapy side effects were identified via ChatGPT–GPT-4-turbo and Gemini–2.5 Flash, then submitted to eight LLMs (ChatGPT–GPT-4-turbo, Gemini–2.5 Flash, Microsoft Copilot, Perplexity, Chatsonic, Mistral, Meta AI–Llama 4, DeepSeek–R1). Responses were assessed using DISCERN and modified DISCERN instruments (reliability and quality), Patient Education Materials Assessment Tool (PEMAT [understandability and actionability]), Flesch-Reading-Ease-Score (FRES [readability]), misinformation score, citations, and wordcounts. Statistical analysis was done by Scheirer-Ray-Hare-test followed by Dunn’s post-hoc-tests and Bonferroni-Holm correction ( p < 0.05). Results: A total of 40 questions belonging to 12 oral health-related categories were identified. Statistically significant differences between LLMs were found for DISCERN, modified DISCERN, PEMAT-understandability, PEMAT-actionability, FRES, and word counts ( p < 0.001). Median DISCERN and modified DISCERN scores amounted from 47.0 (ChatGPT–GPT-4-turbo) to 59.0 (Perplexity, Chatsonic) and from 2.0 (Gemini–2.5 Flash, Mistral) to 5.0 (Perplexity) indicating good to fair reliability. LLMs were understandable (median PEMAT-understandability scores ≥ 75.0), but provided limited specific guidance (median PEMAT-actionability scores ≤ 40) and used complex language (median FRES ≤ 40.2). Misinformation risk was generally low and not statistically significant among LLMs ( p = 0.768). Conclusion: Despite a low overall misinformation risk, deficits in actionability highlight the need for cautious integration of LLMs into HNC patient education.Keywords
Funding Information
- Herzzentrum Göttingen
This publication has 55 references indexed in Scilit:
- Readability, understandability, and quality of online education materials and large language models for retrograde cricopharyngeal muscle dysfunctionEuropean Archives of Oto-Rhino-Laryngology, 2025
- Reporting Guideline for Chatbot Health Advice StudiesJAMA Network Open, 2025
- Daith Piercing, a Social Media Hype on Youtube for the Treatment of Migraine? A Systematic Video AnalysisHealth Science Reports, 2025
- Generative artificial intelligence as a source of breast cancer information for patients: Proceed with cautionCancer, 2024
- Evaluation of Oropharyngeal Cancer Information from Revolutionary Artificial Intelligence ChatbotThe Laryngoscope, 2023
- Assessment of Artificial Intelligence Chatbot Responses to Top Searched Queries About CancerJAMA Oncology, 2023
- Role of Chat GPT in Public HealthAnnals of Biomedical Engineering, 2023
- Squamous cell carcinoma of the oral cavity, larynx, oropharynx and hypopharynx: EHNS–ESMO–ESTRO Clinical Practice Guidelines for diagnosis, treatment and follow-upAnnals of Oncology, 2020
- MASCC/ISOO clinical practice guidelines for the management of mucositis secondary to cancer therapyCancer, 2020
- Education and support needs in patients with head and neck cancer: A multi‐institutional surveyCancer, 2017