Analysis of the Validity of ChatGPT in the Assessment of Perceived Stress and Psychological Distress Among University Students: A Cross-Sectional Study
Authors:
- 9 July 2026
- journal article
- Published by Springer Nature in Psychiatric Quarterly
Abstract
This study assessed the validity of the ChatGPT in measuring perceived stress and psychological distress in college students by comparing the consistency of the ChatGPT's scale-based adaptation of the scenario-based interview question scores with the scores of a validated questionnaire. ChatGPT-4 was used to adapt the Perceived Stress Scale (PSS-4) and the 10-item Kessler Psychological Distress Scale (K10) into scenario-based interview questions with automated scoring. A total of 199 college students completed both the ChatGPT-4-generated scenario questions and the corresponding validated questionnaires. Internal consistency, Spearman correlations, intraclass correlation coefficients (ICCs), Bland-Altman analyses, item-level correlations, and K10 threshold-based agreement were evaluated. All participants completed both assessments, with no missing item or total-score data. ChatGPT-4-generated total scores showed moderate correlations with questionnaire scores for both K10 and PSS-4 (Spearman's ρ = 0.57 for both). The ICC indicated excellent agreement for K10 (ICC = 0.91, 95% CI: 0.89-0.93) and moderate agreement for PSS-4 (ICC = 0.70, 95% CI: 0.63-0.76). Bland-Altman analyses showed small positive mean differences, and item-level correlations were heterogeneous. K10 threshold-based analyses suggested more conservative classification at higher cutoffs. This study provides preliminary feasibility evidence for further validating ChatGPT-4-assisted contextualized scoring as an adjunctive assessment approach. Threshold-based screening performance, reproducibility, criterion validity, and generalizability require further investigation.Keywords
Funding Information
- Hubei Provincial Natural Science Foundation and Traditional Chinese Medicine Innovation and Development Joint Fund (2023AFD160)
This publication has 32 references indexed in Scilit:
- The impact of emotional expression by artificial intelligence recommendation chatbots on perceived humanness and social interactivityDecision Support Systems, 2024
- A critical assessment of using ChatGPT for extracting structured data from clinical notesnpj Digital Medicine, 2024
- The future landscape of large language models in medicineCommunications Medicine, 2023
- Assessment of depression and anxiety in young and old with a question-based computational language approachnpj Mental Health Research, 2023
- Association of Perceived Stress Levels With Long-term Mortality in Patients With Peripheral Artery DiseaseJAMA Network Open, 2020
- Physical injury, health, and well-being: Role of stress perceptionInjury, 2018
- A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability ResearchJournal of Chiropractic Medicine, 2016
- Review of the Psychometric Evidence of the Perceived Stress ScaleAsian Nursing Research, 2012
- Short screening scales to monitor population prevalences and trends in non-specific psychological distressPsychological Medicine, 2002
- Cognitive Theories of Stress and the Issue of Circularity1986