Evaluating the Efficacy of AI-Based Interactive Assessments Using Large Language Models for Depression Screening: Development and Usability Study
Top Cited Papers
Open Access
- 13 January 2026
- journal article
- Published by JMIR Publications Inc. in JMIR Formative Research
- Vol. 10, e78401
Abstract
Background: The evolution of language models, particularly large language models, has introduced transformative potential for psychological assessment, challenging traditional rating scale methods that have dominated clinical practice for over a century. Objective: This study aimed to develop and validate an automated assessment paradigm that integrates natural language processing with conventional measurement tools to assess depressive symptoms, exploring its feasibility as a novel approach in psychological evaluation. Methods: A cohort of 115 participants, including 28 (24.3%) individuals diagnosed with depression, completed the Beck Depression Inventory Fast Screen via a custom ChatGPT interface (BDI-FS-GPT) and the Chinese version of the Patient Health Questionnaire–9 (PHQ-9). Statistical analyses included the Spearman correlation (PHQ-9 vs BDI-FS-GPT scores), Cohen κ (diagnostic agreement), and area under the curve (AUC) evaluation. Results: Spearman analysis revealed a moderate correlation between PHQ-9 and BDI-FS-GPT scores. The Cohen κ indicated moderate diagnostic agreement between the PHQ-9 and the BDI-FS-GPT (κ=0.43; 76.5% agreement), substantial agreement between the BDI-FS-GPT and the clinical diagnosis (κ=0.72; 88.7% agreement), and moderate agreement between the PHQ-9 and the clinical diagnosis (κ=0.55; 71.4% agreement). The BDI-FS-GPT demonstrated excellent diagnostic accuracy (AUC=0.953) at a cutoff of 3, detecting 89.3% of participants with depression with an 11.5% false-positive rate compared to the PHQ-9 (AUC=0.859) at a cutoff of 5 (sensitivity=71.4%; false-positive rate=13.8%). Participants also reported significantly higher satisfaction with the automated assessment compared to the traditional scale (P=.02). Conclusions: The automated assessment paradigm framework combines the interactivity and personalization of natural language processing–powered tools with the psychometric rigor of traditional scales, suggesting a preliminary feasibility paradigm for future psychological assessment. Its ability to enhance engagement while maintaining reliability and validity provides encouraging evidence, warranting validation in larger and more diverse studies as large language model technology advances. International Registered Report Identifier (IRRID): RR2-10.1101/2024.07.19.24310543Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- Beyond rating scales: With targeted evaluation, large language models are poised for psychological assessmentPsychiatry Research, 2023
- Natural language analyzed with AI-based transformers predict traditional subjective well-being measures approaching the theoretical upper limits in accuracyScientific Reports, 2022
- World Trade Center responders in their own words: predicting PTSD symptom trajectories with AI-based language analyses of interviewsPsychological Medicine, 2021
- Semantic measures: Using natural language processing to measure, differentiate, and describe psychological constructs.Psychological Methods, 2019
- Psychometric Evaluation of the USE (Usefulness, Satisfaction, and Ease of use) Questionnaire for Reliability and ValidityProceedings of the Human Factors and Ergonomics Society Annual Meeting, 2018
- Reliability and validity of the beck depression inventory-fast screen for medical patients in the general German populationJournal of Affective Disorders, 2014
- Assessing depression in patients with chronic pain: A comparison of three rating scalesJournal of Affective Disorders, 2011
- Reliability and validity of the Hospital Anxiety and Depression Scale and the Beck Depression Inventory (Full and FastScreen scales) in detecting depression in persons with hepatitis CJournal of Affective Disorders, 2007
- Sample size requirements for the design of reliability study: review and new resultsStatistical Methods in Medical Research, 2004
- The PHQ-9Journal of General Internal Medicine, 2001