Julia Thomas

Large Language Models’ (LLMs) potential for psychological diagnostics requires systematic evaluation. We aimed to investigate conditions for reliable and valid psychological assessments, focusing on suicide risk evaluation in clinical data by comparing LLM-generated ratings with human expert ratings across model configurations., analyzing 100 youth crisis text line conversation transcripts rated by four experts using the Nurses’ Global Assessment of Suicide Scale (NGASR). Using Mixtral-8x7B-Instruct, we generated ratings across three temperature settings and prompting styles (zero-shot, few-shot, chain-of-thought). Across configurations we compared (a) inter-rating-reliability for AI-generated NGASR risk and sum scores, (b) LLM-to-human observer agreement regarding sum score, risk category, and item, using Krippendorff’s α, (c) classification metrics of risk categories and individual items against human ratings. LLM configuration strongly influenced assessment reliability. Zero-shot prompting at temperature 0 yielded perfect inter-rating reliability (α = 1.00, 95% CI: [1–1] for high & very high risk), while few-shot prompting showed best human-AI agreement for very high risk (α = 0.78, 95% CI: [0.67–0.89]) and strongest classification performance (balanced accuracy 0.54–0.71). Lower temperatures consistently improved reliability and accuracy. However, critical clinical items showed poor validity. Our findings establish optimal conditions (zero temperature, task-specific prompting) for LLM-based psychological assessment. However, inconsistent clinical item performance and only moderate LLM-to-human observer agreement limit LLMs to initial screening rather than detailed assessment, requiring careful parameter control and validation.

Background : Large Language Models´ (LLMs) potential for psychological diagnostics requires systematic evaluation. Objective : To investigate conditions for reliable and valid psychological assessments, focusing on suicide risk evaluation in clinical data by comparing LLM-generated ratings with human expert ratings across across configurations. Methods : We analyzed 100 youth crisis conversation transcripts rated by four experts using the Nurses Global Assessment of Suicide Scale (NGASR). Using Mixtral-7x8b-Instruct, we generated ratings across three temperature settings and prompting styles (zero-shot, few-shot, chain-of-thought). Across configurations we compared a) inter-rating-reliability for AI-generated NGASR risk and sum scores, b) LLM-to-human observer agreement regarding sum score, risk category, and item, using Krippendorff´s α, c) classification metrics of risk categories and individual items against human ratings. Results : LLM configuration strongly influenced assessment reliability. Zero-shot prompting at temperature 0 yielded perfect inter-rating reliability (α=1.00, 95% CI: [1-1] for high & very high risk), while few-shot prompting showed best human-AI agreement for very high risk (α=0.78, 95% CI: [0.67-0.89]) and strongest classification performance (balanced accuracy 0.54-0.71). Lower temperatures consistently improved reliability and accuracy. However, critical clinical items showed poor validity. Discussion : Our findings establish optimal conditions (zero temperature, task-specific prompting) for LLM-based psychological assessment. However, inconsistent clinical item performance and only moderate to-human agreement limit LLMs to initial screening rather than detailed assessment, requiring careful parameter control and validation.

Background: Suicide represents a critical public health concern, and machine learning (ML) models offer the potential for identifying at-risk individuals. Recent studies using benchmark datasets and real-world social media data have demonstrated the capability of pretrained large language models in predicting suicidal ideation and behaviors (SIB) in speech and text. Objective: This study aimed to (1) develop and implement ML methods for predicting SIBs in a real-world crisis helpline dataset, using transformer-based pretrained models as a foundation; (2) evaluate, cross-validate, and benchmark the model against traditional text classification approaches; and (3) train an explainable model to highlight relevant risk-associated features. Methods: We analyzed chat protocols from adolescents and young adults (aged 14-25 years) seeking assistance from a German crisis helpline. An ML model was developed using a transformer-based language model architecture with pretrained weights and long short-term memory layers. The model predicted suicidal ideation (SI) and advanced suicidal engagement (ASE), as indicated by composite Columbia-Suicide Severity Rating Scale scores. We compared model performance against a classical word-vector-based ML model. We subsequently computed discrimination, calibration, clinical utility, and explainability information using a Shapley Additive Explanations value-based post hoc estimation model. Results: The dataset comprised 1348 help-seeking encounters (1011 for training and 337 for testing). The transformer-based classifier achieved a macroaveraged area under the curve (AUC) receiver operating characteristic (ROC) of 0.89 (95% CI 0.81-0.91) and an overall accuracy of 0.79 (95% CI 0.73-0.99). This performance surpassed the word-vector-based baseline model (AUC-ROC=0.77, 95% CI 0.64-0.90; accuracy=0.61, 95% CI 0.61-0.80). The transformer model demonstrated excellent prediction for nonsuicidal sessions (AUC-ROC=0.96, 95% CI 0.96-0.99) and good prediction for SI and ASE, with AUC-ROCs of 0.85 (95% CI 0.97-0.86) and 0.87 (95% CI 0.81-0.88), respectively. The Brier Skill Score indicated a 44% improvement in classification performance over the baseline model. The Shapley Additive Explanations model identified language features predictive of SIBs, including self-reference, negation, expressions of low self-esteem, and absolutist language. Conclusions: Neural networks using large language model–based transfer learning can accurately identify SI and ASE. The post hoc explainer model revealed language features associated with SI and ASE. Such models may potentially support clinical decision-making in suicide prevention services. Future research should explore multimodal input features and temporal aspects of suicide risk.

BACKGROUND Suicide represents a critical public health concern, and machine learning (ML) models offer the potential for identifying at-risk individuals. Recent studies using benchmark datasets and real-world social media data have demonstrated the capability of pretrained large language models in predicting suicidal ideation and behaviors (SIB) in speech and text. OBJECTIVE This study aimed to (1) develop and implement ML methods for predicting SIBs in a real-world crisis helpline dataset, using transformer-based pretrained models as a foundation; (2) evaluate, cross-validate, and benchmark the model against traditional text classification approaches; and (3) train an explainable model to highlight relevant risk-associated features. METHODS We analyzed chat protocols from adolescents and young adults (aged 14-25 years) seeking assistance from a German crisis helpline. An ML model was developed using a transformer-based language model architecture with pretrained weights and long short-term memory layers. The model predicted suicidal ideation (SI) and advanced suicidal engagement (ASE), as indicated by composite Columbia-Suicide Severity Rating Scale scores. We compared model performance against a classical word-vector-based ML model. We subsequently computed discrimination, calibration, clinical utility, and explainability information using a Shapley Additive Explanations value-based post hoc estimation model. RESULTS The dataset comprised 1348 help-seeking encounters (1011 for training and 337 for testing). The transformer-based classifier achieved a macroaveraged area under the curve (AUC) receiver operating characteristic (ROC) of 0.89 (95% CI 0.81-0.91) and an overall accuracy of 0.79 (95% CI 0.73-0.99). This performance surpassed the word-vector-based baseline model (AUC-ROC=0.77, 95% CI 0.64-0.90; accuracy=0.61, 95% CI 0.61-0.80). The transformer model demonstrated excellent prediction for nonsuicidal sessions (AUC-ROC=0.96, 95% CI 0.96-0.99) and good prediction for SI and ASE, with AUC-ROCs of 0.85 (95% CI 0.97-0.86) and 0.87 (95% CI 0.81-0.88), respectively. The Brier Skill Score indicated a 44% improvement in classification performance over the baseline model. The Shapley Additive Explanations model identified language features predictive of SIBs, including self-reference, negation, expressions of low self-esteem, and absolutist language. CONCLUSIONS Neural networks using large language model–based transfer learning can accurately identify SI and ASE. The post hoc explainer model revealed language features associated with SI and ASE. Such models may potentially support clinical decision-making in suicide prevention services. Future research should explore multimodal input features and temporal aspects of suicide risk.

Background: Text messaging is widely used by young people for communicating and seeking mental health support through chat-based helplines. However, written communication lacks nonverbal cues, and language usage is an important source of information about a person’s mental health state and is known to be a marker for psychopathology. Objective: The aim of the study was to investigate language usage, and its gender differences and associations with the presence of psychiatric symptoms within a chat counseling service for adolescents and young adults. Methods: For this study, the anonymized chat content of a German messenger–based psychosocial chat counseling service for children and adolescents (“krisenchat”) between May 2020 and July 2021 was analyzed. In total, 661,131 messages from 6962 users were evaluated using Linguistic Inquiry and Word Count, considering the following linguistic variables: first-person singular and plural pronouns, negations, positive and negative emotion words, insight words, and causation words. Descriptive analyses were performed, and gender differences of those variables were evaluated. Finally, a binary logistic regression analysis examined the predictive value of linguistic variables on the presence of psychiatric symptoms. Results: Across all analyzed chats, first-person singular pronouns were used most frequently (965,542/8,328,309, 11.6%), followed by positive emotion words (408,087/8,328,309, 4.9%), insight words (341,460/8,328,309, 4.1%), negations (316,475/8,328,309, 3.8%), negative emotion words (266,505/8,328,309, 3.2%), causation words (241,520/8,328,309, 2.9%), and first-person plural pronouns (499,698/8,328,309, 0.6%). Female users and users identifying as diverse used significantly more first-person singular pronouns and insight words than male users (both P<.001). Negations were significantly more used by female users than male users or users identifying as diverse (P=.007). Similar findings were noted for negative emotion words (P=.01). The regression model of predicting psychiatric symptoms by linguistic variables was significant and indicated that increased use of first-person singular pronouns (odds ratio [OR] 1.05), negations (OR 1.11), and negative emotion words (OR 1.15) was positively associated with the presence of psychiatric symptoms, whereas increased use of first-person plural pronouns (OR 0.39) and causation words (OR 0.90) was negatively associated with the presence of psychiatric symptoms. Suicidality, self-harm, and depression showed the most significant correlations with linguistic variables. Conclusions: This study highlights the importance of examining linguistic features in chat counseling contexts. By integrating psycholinguistic findings into counseling practice, counselors may better understand users’ psychological processes and provide more targeted support. For instance, certain linguistic features, such as high use of first-person singular pronouns, negations, or negative emotion words, may indicate the presence of psychiatric symptoms, particularly among female users and users identifying as diverse. Further research is needed to provide an in-depth look into language processes within chat counseling services.

BACKGROUND Text messaging is widely used by young people for communicating and seeking mental health support through chat-based helplines. However, written communication lacks nonverbal cues, and language usage is an important source of information about a person’s mental health state and is known to be a marker for psychopathology. OBJECTIVE The aim of the study was to investigate language usage, and its gender differences and associations with the presence of psychiatric symptoms within a chat counseling service for adolescents and young adults. METHODS For this study, the anonymized chat content of a German messenger–based psychosocial chat counseling service for children and adolescents (“krisenchat”) between May 2020 and July 2021 was analyzed. In total, 661,131 messages from 6962 users were evaluated using Linguistic Inquiry and Word Count, considering the following linguistic variables: first-person singular and plural pronouns, negations, positive and negative emotion words, insight words, and causation words. Descriptive analyses were performed, and gender differences of those variables were evaluated. Finally, a binary logistic regression analysis examined the predictive value of linguistic variables on the presence of psychiatric symptoms. RESULTS Across all analyzed chats, first-person singular pronouns were used most frequently (965,542/8,328,309, 11.6%), followed by positive emotion words (408,087/8,328,309, 4.9%), insight words (341,460/8,328,309, 4.1%), negations (316,475/8,328,309, 3.8%), negative emotion words (266,505/8,328,309, 3.2%), causation words (241,520/8,328,309, 2.9%), and first-person plural pronouns (499,698/8,328,309, 0.6%). Female users and users identifying as diverse used significantly more first-person singular pronouns and insight words than male users (both P<.001). Negations were significantly more used by female users than male users or users identifying as diverse (P=.007). Similar findings were noted for negative emotion words (P=.01). The regression model of predicting psychiatric symptoms by linguistic variables was significant and indicated that increased use of first-person singular pronouns (odds ratio [OR] 1.05), negations (OR 1.11), and negative emotion words (OR 1.15) was positively associated with the presence of psychiatric symptoms, whereas increased use of first-person plural pronouns (OR 0.39) and causation words (OR 0.90) was negatively associated with the presence of psychiatric symptoms. Suicidality, self-harm, and depression showed the most significant correlations with linguistic variables. CONCLUSIONS This study highlights the importance of examining linguistic features in chat counseling contexts. By integrating psycholinguistic findings into counseling practice, counselors may better understand users’ psychological processes and provide more targeted support. For instance, certain linguistic features, such as high use of first-person singular pronouns, negations, or negative emotion words, may indicate the presence of psychiatric symptoms, particularly among female users and users identifying as diverse. Further research is needed to provide an in-depth look into language processes within chat counseling services.

Background: Mental crises have high prevalences in adolescence. Early interventions appear to be highly important to diminish the risk of the deterioration, recurrence, or chronification of symptoms. In recent years, various providers have started offering live chat support in psychological crises. The messenger-based psychological counseling service krisenchat aims to support young people in crises and, if necessary, provide a recommendation for a referral to the health care system or to seek further help from a trusted adult person. Objective: This study aimed to investigate the impact of using the counseling service of krisenchat on the further help-seeking behavior of young people, and to identify associated factors of further help-seeking. Methods: This longitudinal study analyzed anonymous data from 247 individuals who used krisenchat between October 2021 and March 2022, and received a recommendation for further help-seeking. An online survey directly after the chat assessed the perceived helpfulness of the chat and well-being after the chat. After 4 weeks, further help-seeking, facilitators and barriers to help-seeking, and self-efficacy were assessed in an online follow-up survey. Results: The most frequently recommended services or persons to seek further help from included a psychotherapist or social psychiatric service (75/225, 33.3%), a school psychologist or school social worker (52/225, 23.1%), and the user’s parents (45/225, 20.0%). Of the 247 users, 120 (48.6%) indicated that they contacted the recommended service or person, and of these, 87 (72.5%) stated that they already had an appointment (or talk) with the respective service or person or that an appointment (or talk) was scheduled. The most frequently reported facilitators for further help-seeking were mental health literacy (54/120, 45.0%), improvement of self-efficacy (55/120, 45.8%), and symptom recognition (40/120, 33.3%). In users not displaying further help-seeking behavior, the most frequent barriers included stigmatization (60/127, 47.2%), lack of mental health literacy (59/127, 46.5%), need for self-reliance and autonomy (53/127, 41.7%), and negative family beliefs regarding help services (53/127, 41.7%). Subgroup comparisons indicated significantly higher levels of self-efficacy in users displaying further help-seeking behavior than in those not displaying further help-seeking behavior. Both subgroups did not differ in gender, age, recommended service or person, chat topics, perceived helpfulness, and well-being. Conclusions: The findings of this study indicate that children and young adults receiving counseling on krisenchat benefit in terms of seeking further help. Further help-seeking seems to be associated with higher levels of self-efficacy. Trial Registration: Deutsches Register Klinischer Studien DRKS00026671; https://tinyurl.com/4fm5xe68

BACKGROUND Mental crises have high prevalences in adolescence. Early interventions appear to be highly important to diminish the risk of the deterioration, recurrence, or chronification of symptoms. In recent years, various providers have started offering live chat support in psychological crises. The messenger-based psychological counseling service krisenchat aims to support young people in crises and, if necessary, provide a recommendation for a referral to the health care system or to seek further help from a trusted adult person. OBJECTIVE This study aimed to investigate the impact of using the counseling service of krisenchat on the further help-seeking behavior of young people, and to identify associated factors of further help-seeking. METHODS This longitudinal study analyzed anonymous data from 247 individuals who used krisenchat between October 2021 and March 2022, and received a recommendation for further help-seeking. An online survey directly after the chat assessed the perceived helpfulness of the chat and well-being after the chat. After 4 weeks, further help-seeking, facilitators and barriers to help-seeking, and self-efficacy were assessed in an online follow-up survey. RESULTS The most frequently recommended services or persons to seek further help from included a psychotherapist or social psychiatric service (75/225, 33.3%), a school psychologist or school social worker (52/225, 23.1%), and the user’s parents (45/225, 20.0%). Of the 247 users, 120 (48.6%) indicated that they contacted the recommended service or person, and of these, 87 (72.5%) stated that they already had an appointment (or talk) with the respective service or person or that an appointment (or talk) was scheduled. The most frequently reported facilitators for further help-seeking were mental health literacy (54/120, 45.0%), improvement of self-efficacy (55/120, 45.8%), and symptom recognition (40/120, 33.3%). In users not displaying further help-seeking behavior, the most frequent barriers included stigmatization (60/127, 47.2%), lack of mental health literacy (59/127, 46.5%), need for self-reliance and autonomy (53/127, 41.7%), and negative family beliefs regarding help services (53/127, 41.7%). Subgroup comparisons indicated significantly higher levels of self-efficacy in users displaying further help-seeking behavior than in those not displaying further help-seeking behavior. Both subgroups did not differ in gender, age, recommended service or person, chat topics, perceived helpfulness, and well-being. CONCLUSIONS The findings of this study indicate that children and young adults receiving counseling on krisenchat benefit in terms of seeking further help. Further help-seeking seems to be associated with higher levels of self-efficacy. CLINICALTRIAL Deutsches Register Klinischer Studien DRKS00026671; https://tinyurl.com/4fm5xe68