Evaluating the accuracy of ChatGPT model versions for giving care-seeking advice
Authors:
Top Cited Papers
Open Access
- 25 February 2026
- journal article
- research article
- Published by Springer Nature in Communications Medicine
- Vol. 6 (1)
Abstract
Background: Artificial Intelligence tools such as ChatGPT are increasingly used by laypeople to support their care-seeking decisions, although the accuracy of newer models remains unclear. We aimed to evaluate the accuracy of care-seeking advice that is generated by all currently available ChatGPT models. Methods: We evaluated 22 ChatGPT models using 45 validated vignettes, each prompted ten times (9,900 total assessments). Each model classified the vignettes as requiring emergency care, non-emergency care, or self-care. We evaluated accuracy against each case’s gold standard solution (determined by two physicians), examined the variability across trials, and tested algorithms to aggregate multiple recommendations to improve accuracy. Results: We show that o1-mini achieves the highest accuracy (74%), but we cannot observe an overall improvement with newer models – although reasoning models (e.g., o4-mini) improved their accuracy in identifying self-care cases. Selecting the lowest urgency level across multiple trials improves accuracy by 4 percentage points. Conclusions: Although newer increasingly provide self-care advice, their accuracy remains insufficient for standalone use. However, making use of output variability with aggregation algorithms can improve the performance of existing models.Keywords
This publication has 46 references indexed in Scilit:
- Testing and Evaluation of Health Care Applications of Large Language ModelsJAMA, 2025
- The RepVig framework for designing use-case specific representative vignettes and evaluating triage accuracy of laypeople and symptom assessment applicationsScientific Reports, 2024
- Large Language Model Influence on Diagnostic ReasoningJAMA Network Open, 2024
- The diagnostic and triage accuracy of the GPT-3 artificial intelligence model: an observational studyThe Lancet Digital Health, 2024
- ChatGPT can help guide and empower patients after prostate cancer diagnosisProstate Cancer and Prostatic Diseases, 2024
- ChatGPT and Generating a Differential Diagnosis Early in an Emergency Department PresentationAnnals of Emergency Medicine, 2023
- The Challenges for Regulating Medical Use of ChatGPT and Other Large Language ModelsJAMA, 2023
- Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media ForumJAMA Internal Medicine, 2023
- Evidence for overuse of medical services around the worldThe Lancet, 2017
- Comparison of Physician and Computer Diagnostic AccuracyJAMA Internal Medicine, 2016