Are clinical improvements in large language models a reality? Longitudinal comparisons of ChatGPT models and DeepSeek-R1 for psychiatric assessments and interventions
- 31 July 2025
- journal article
- Published by SAGE Publications in International Journal of Social Psychiatry
- Vol. 72 (1) , 91-102
Abstract
Background: Potential clinical applications for emerging large-language models (LLMs; e.g. ChatGPT) are well-documented, and newer systems (e.g. DeepSeek) have attracted increasing attention. Yet, important questions endure about their reliability and cultural responsiveness in psychiatric settings. Methods: This study explored the diagnostic accuracy, therapeutic appropriateness and cultural sensitivity of ChatGPT-4o, ChatGPT-4.5, and DeepSeek-R1 (all March 2025 versions). DeepSeek-R1 was evaluated for one of the first times in this context, and this also marks one of the first longitudinal inquiries into LLMs in psychiatry. Three psychiatric cases from earlier literature about sleep-related problems and cooccurring issues were utilised, allowing for cross-comparisons with a 2023 ChatGPT version, alongside culturally-specific vignette adaptations. Thus, overall, outputs for six scenarios were derived and were subsequently qualitatively reviewed by four psychiatrists for their strengths and limitations. Results: ChatGPT-4o, ChatGPT-4.5, and DeepSeek-R1 showed modest improvements from the 2023 ChatGPT model but still exhibited significant limitations. Communication was empathetic and non-pharmacological advice typically adhered to evidence-based practices. Primary diagnoses were broadly accurate but often omitted somatic factors and comorbidities. Nevertheless, consistent with past findings, clinical reasoning worsened as case complexity increased; this was especially apparent for suicidality safeguards and risk stratification. Pharmacological recommendations frequently diverged from established guidelines, whilst cultural adaptations remained largely superficial. Finally, output variance was noted in several cases, and the LLMs occasionally failed to clarify their inability to prescribe medication. Conclusion: Despite incremental advancements, ChatGPT-4o, ChatGPT-4.5 and DeepSeek-R1 were affected by major shortcomings, particularly in risk evaluation, evidence-based practice adherence, and cultural awareness. Presently, we conclude that these tools cannot substitute mental health professionals but may confer adjunctive benefits. Notably, DeepSeek-R1 did not fall behind its counterparts, warranting further inquiries in jurisdictions permitting its use. Equally, greater emphasis on transparency and prompt engineering would also be necessary for safe and equitable LLM deployment in psychiatry.Keywords
This publication has 37 references indexed in Scilit:
- Appraising the performance of ChatGPT in psychiatry using 100 clinical case vignettesAsian Journal of Psychiatry, 2023
- World Psychiatric Association-Asian Journal of Psychiatry Commission on Psychiatric Education in the 21st centuryAsian Journal of Psychiatry, 2023
- From ELIZA to ChatGPTResonance, 2023
- Effect of brexanolone on depressive symptoms, anxiety, and insomnia in women with postpartum depression: Pooled analyses from 3 double-blind, randomized, placebo-controlled clinical trials in the HUMMINGBIRD clinical programJournal of Affective Disorders, 2022
- Daridorexant: First ApprovalDrugs, 2022
- Antidepressant treatment for postnatal depressionCochrane Database of Systematic Reviews, 2021
- Perinatal Insomnia and Mental Health: a Review of Recent LiteratureCurrent Psychiatry Reports, 2020
- Perinatal mental health: a review of progress and challengesWorld Psychiatry, 2020
- Sleep and rheumatic diseasesBest Practice & Research Clinical Rheumatology, 2019
- Cultural influences in mental health treatmentCurrent Opinion in Psychology, 2015