AI health advice accuracy varies across languages and contexts.
Authors:
Preprint
- 9 September 2025
- preprint
- Published by Springer Nature in Research Square
Abstract
Using basic health statements authorized by UK and EU registers and ~9,100 journalist-vetted public-health assertions on topics such as abortion, COVID-19 and politics from sources ranging from peer-reviewed journals and government advisories to social media and news across the political spectrum, we benchmark seven leading large language models in 21 languages. We find that, despite high accuracy on English-centric textbook claims, performance falls in multiple non-European languages and fluctuates by topic and source. This highlights the urgency of comprehensive multilingual, domain-aware validation before deploying AI in global health communication.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Use of ChatGPT to obtain health information in Australia, 2024: insights from a nationally representative surveyMedical Journal of Australia, 2025
- Toward expert-level medical question answering with large language modelsNature Medicine, 2025
- ChatGPT May Improve Access to Language-Concordant Care for Patients With Non–English Language PreferencesJMIR Medical Education, 2024
- Generative Artificial Intelligence to Transform Inpatient Discharge Summaries to Patient-Friendly Language and FormatJAMA Network Open, 2024
- Large language models encode clinical knowledgeNature, 2023
- Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language modelsPLOS Digital Health, 2023
- A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity2023