Evaluation of ChatGPT for NLP-based Mental Health Applications
Preprint
Abstract
Large language models (LLM) have been successful in several natural language understanding tasks and could be relevant for natural language processing (NLP)-based mental health application research. In this work, we report the performance of LLM-based ChatGPT (with gpt-3.5-turbo backend) in three text-based mental health classification tasks: stress detection (2-class classification), depression detection (2-class classification), and suicidality detection (5-class classification). We obtained annotated social media posts for the three classification tasks from public datasets. Then ChatGPT API classified the social media posts with an input prompt for classification. We obtained F1 scores of 0.73, 0.86, and 0.37 for stress detection, depression detection, and suicidality detection, respectively. A baseline model that always predicted the dominant class resulted in F1 scores of 0.35, 0.60, and 0.19. The zero-shot classification accuracy obtained with ChatGPT indicates a potential use of language models for mental health classification tasks.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Large language models encode clinical knowledgeNature, 2023
- A deep learning model for detecting mental illness from user content on social mediaScientific Reports, 2020
- Methods in predictive techniques for mental health status on social media: a critical reviewnpj Digital Medicine, 2020
- Detection of Depression-Related Posts in Reddit Social Media ForumIEEE Access, 2019
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Dreaddit: A Reddit Dataset for Stress Analysis in Social Media2019
- Question Answering for Suicide Risk Assessment Using Reddit2019
- Characterisation of mental health conditions in social media using Informed Deep LearningScientific Reports, 2017