Performance of Large Language Models in Nursing Licensure Examinations: A Systematic Review and Meta-Analysis
Authors:
Preprint
- 23 January 2026
- preprint
- Published by Springer Nature in Research Square
Abstract
Objectives : This systematic review and meta-analysis assessed the performance of large language models (LLMS) in nursing licensure examinations. Despite the increasing use of LLMS in healthcare education, their capabilities in nursing licensure examinations remain uncertain. This study provides evidence on the accuracy and limitations of LLMs to help guide their integration into nursing education and licensure. Design : The systematic review and meta-analysis adhered to PRISMA 2020 guidelines. Data sources : PubMed, CINAHL, PsycINFO, EMCARE, and ERIC were searched from April to June 2025. Eligibility criteria: Studies were eligible if they evaluated LLMs (e.g., GPT-4, ChatGPT, Qwen-2.5) using multiple-choice nursing licensure questions under exam-like conditions and reported quantitative accuracy. Open-ended items were excluded from the meta-analysis but narratively synthesised Review methods : Two reviewers independently screened, extracted data, and appraised the risk of bias. A random-effects meta-analysis estimated pooled accuracy; subgroup and meta-regression analyses explored heterogeneity. Results : Twelve studies assessed 13,870 MCQs across five exam systems and eight LLMs. Pooled accuracy was 69.6% (95% CI: 65.6–73.6%) with substantial heterogeneity (I² = 98%). GPT-4 outperformed GPT-3.5 (77.2% vs. 60.4%, six studies); domain-customised and newer models reached 93.6%. LLMs excelled in general medicine and pharmacology but underperformed in ethics and psychosocial integrity. Accuracy differed significantly by exam system (p < 0.01), but not by question difficulty (p = 0.90) or format (p = 0.96). Translated NCLEX-RN items reduced accuracy (p = 0.03); CNNLE was the only system with a significant positive effect (p < 0.001). Methodological variability and underreporting of model parameters were common. Conclusions: LLMs show promise for low-stakes educational applications, such as formative assessments within hybrid teaching models; however, they are unsuitable for unmoderated, high-stakes licensure decisions due to inconsistent performance. Regulatory guidelines, equitable access, and nursing-specific model development are needed to ensure fairness and validity. Research must prioritise standardised frameworks, error analysis, and broader geographic representation to address these limitations.Keywords
All Related Versions
This publication has 37 references indexed in Scilit:
- Patterns, advances, and gaps in using ChatGPT and similar technologies in nursing education: A PAGER scoping reviewNurse Education Today, 2025
- Large Language Models (LLMs) and Generative Artificial Intelligence (GenAI)2025
- Deriving insights from enhanced accuracy: Leveraging prompt engineering in custom GPT for assessing Chinese Nursing Licensing ExamNurse Education in Practice, 2025
- The performance of ChatGPT versus neurosurgery residents in neurosurgical board examination-like questions: a systematic review and meta-analysisNeurosurgical Review, 2024
- Artificial intelligence in education: A systematic literature reviewExpert Systems with Applications, 2024
- ChatGPT in radiology: A systematic review of performance, pitfalls, and future perspectivesDiagnostic and Interventional Imaging, 2024
- A rapid review on current and potential uses of large language models in nursingInternational Journal of Nursing Studies, 2024
- Evaluation of ChatGPT-generated medical responses: A systematic review and meta-analysisJournal of Biomedical Informatics, 2024
- A systematic review and meta-analysis on ChatGPT and its utilization in medical and dental researchHeliyon, 2023
- The value of a second reviewer for study selection in systematic reviewsResearch Synthesis Methods, 2019