Comparing the Neuropsychology Knowledge Base of Publicly Available Large Language Models
- 8 December 2025
- journal article
- Published by Oxford University Press (OUP) in Archives of Clinical Neuropsychology
- Vol. 41 (3)
Abstract
Objective: Assess the neuropsychology knowledge base of open-access large language models (LLMs) and inform potential applications in the field. Method: We obtained 600 multiple-choice practice questions from the “Be Ready for ABPP in Neuropsychology” website of the American Academy of Clinical Neuropsychology. We tested OpenAI (GPT-3.5, GPT-4, o3-mini-high) and Google (Gemini 1.0, 2.0 Flash Thinking Experimental [FTE]) models in two trials (T1: used 20-question blocks, T2: single-item re-administration of incorrect questions during T1). We compared AI-estimated to actual accuracy using a paired-samples t-test, while binomial logit generalized linear mixed-effects models (GLMMs) with a random intercept for item compared LLMs and item-level predictors (domain of practice, word count, position-in-block, and higher/lower order question type). Finally, we thematically analyzed the questions missed by the top models. Results: OpenAI o3-mini-high demonstrated the highest accuracy (T1:87.0%, T2:90.3%), followed by Gemini 2.0 FTE (T1:81.7%, T2:88.7%), GPT-4 (T1:74.0%, T2:85.5%), GPT-3.5 (T1:62.5%), and Gemini 1.0 (T1:52.3%). On average, LLMs overestimated their accuracy by 15.8% (87.3% vs. 71.5%, p < .001). In the GLMMs, specific LLM (p < .001) and practice domain (p = .045) were the only significant predictors of accuracy. Chain-of-thought “reasoning” models outperformed older models (p < .001) but displayed inaccuracies pertaining to aspects of neuropsychological testing and interpretation, diagnostic reasoning, clinical decision making, and neuroanatomy/neuroimaging. Conclusions: Chain-of-thought “reasoning” models displayed the highest accuracy, suggesting they may have utility in neuropsychology education, research, and clinical practice. However, the LLMs displayed persistent neuropsychology content area weaknesses and tended to present inaccurate information with confidence, highlighting the need for caution when interpreting LLM output.Keywords
This publication has 32 references indexed in Scilit:
- Towards accurate differential diagnosis with large language modelsNature, 2025
- A systematic review and meta-analysis of diagnostic performance comparison between generative AI and physiciansnpj Digital Medicine, 2025
- Toward expert-level medical question answering with large language modelsNature Medicine, 2025
- A strategy for cost-effective large language model use at health system-scalenpj Digital Medicine, 2024
- A survey on large language model based autonomous agentsFrontiers of Computer Science, 2024
- Adapted large language models can outperform medical experts in clinical text summarizationNature Medicine, 2024
- Beyond rating scales: With targeted evaluation, large language models are poised for psychological assessmentPsychiatry Research, 2023
- Performance of Large Language Models on a Neurology Board–Style ExaminationJAMA Network Open, 2023
- Large language models in medicineNature Medicine, 2023
- A Paradigm Shift from “Human Writing” to “Machine Generation” in Personality Test Development: an Application of State-of-the-Art Natural Language ProcessingJournal of Business and Psychology, 2022