Aiding Large Language Models Using Clinical Scoresheets for Neurobehavioral Diagnostic Classification From Text: Algorithm Development and Validation
Open Access
- 21 October 2025
- journal article
- Published by JMIR Publications Inc. in JMIR AI
- Vol. 4, e75030
Abstract
Background: Large language models (LLMs) have demonstrated the ability to perform complex tasks traditionally requiring human intelligence. However, their use in automated diagnostics for psychiatry and behavioral sciences remains under-studied. Objective: This study aimed to evaluate whether incorporating structured clinical assessment scales improved the diagnostic performance of LLM-based chatbots for neuropsychiatric conditions (we evaluated autism spectrum disorder, aphasia, and depression datasets) across two prompting strategies: (1) direct diagnosis and (2) code generation. We aimed to contextualize LLM-based diagnostic performance by benchmarking it against prior work that applied traditional machine learning classifiers to the same datasets, allowing us to assess whether LLMs offer competitive or complementary capabilities in clinical classification tasks. Methods: We tested two approaches using ChatGPT, Gemini, and Claude models: (1) direct diagnostic querying and (2) execution of chatbot-generated code for classification. Three diagnostic datasets were used: ASDBank (autism spectrum disorder), AphasiaBank (aphasia), and Distress Analysis Interview Corpus-Wizard-of-Oz interviews (depression and related conditions). Each approach was evaluated with and without the aid of clinical assessment scales. Performance was compared to existing machine learning benchmarks on these datasets. Results: Across all 3 datasets, incorporating clinical assessment scales led to little improvement in performance, and results remained inconsistent and generally below those reported in previous studies. On the AphasiaBank dataset, the direct diagnosis approach using ChatGPT with GPT-4 produced a low F1-score of 65.6% and specificity of 33%. The code generation method improved results, with ChatGPT with GPT-4o reaching an F1-score of 81.4%, specificity of 78.6%, and sensitivity of 84.3%. ChatGPT with GPT-o3 and Gemini 2.5 Pro performed even better, with F1-scores of 86.5% and 84.3%, respectively. For the ASDBank dataset, direct diagnosis results were lower, with F1-scores of 56% for ChatGPT with GPT-4 and 54% for ChatGPT with GPT-4o. Under code generation, ChatGPT with GPT-o3 reached 67.9%, and Claude 3.5 performed reasonably well with 60%. Gemini 2.5 Pro failed to respond under this assessment condition. In the Distress Analysis Interview Corpus-Wizard-of-Oz dataset, direct diagnosis yielded high accuracy (70.9%) but poor F1-scores of 8% using ChatGPT with GPT-4o. Code generation improved specificity—88.6% with ChatGPT with GPT-4o—but F1-scores remained low overall. These findings suggest that, while clinical scales may help structure outputs, prompting alone remains insufficient for consistent diagnostic accuracy. Conclusions: Current LLM-based chatbots, when prompted naively, underperform on psychiatric and behavioral diagnostic tasks compared to specialized machine learning models. Clinical assessment scales might modestly aid chatbot performance, but more sophisticated prompt engineering and domain integration are likely required to reach clinically actionable standards.Keywords
All Related Versions
This publication has 27 references indexed in Scilit:
- Applying machine learning to language problem analysis2024
- Enhancing Depression Detection in Clinical Interviews: Integration of fused BERT-BiLSTM Model and XGBoost2024
- MindWatch: A Smart Cloud-based AI solution for Suicide Ideation Detection leveraging Large Language ModelsmedRxiv, 2023
- Will Affective Computing Emerge From Foundation Models and General Artificial Intelligence? A First Evaluation of ChatGPTIEEE Intelligent Systems, 2023
- Survey of Hallucination in Natural Language GenerationACM Computing Surveys, 2023
- Potential Use of Chat GPT in Global WarmingAnnals of Biomedical Engineering, 2023
- ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scopeInternet of Things and Cyber-Physical Systems, 2023
- Machine learning and deep learning approach for medical image analysis: diagnosis to detectionMultimedia Tools and Applications, 2022
- Crowdsourced privacy-preserved feature tagging of short home videos for machine learning ASD detectionScientific Reports, 2021
- AphasiaBank: Methods for studying discourseAphasiology, 2011