Automated Safety Testing and Reporting Application for Conversational Safety Monitoring of Generative AI Tools for Mental Health: Development and Validation Study
Open Access
- 19 May 2026
- journal article
- research article
- Published by JMIR Publications Inc. in JMIR Mental Health
- Vol. 13, e91367
Abstract
Background: Artificial intelligence (AI)–based conversational tools are rapidly expanding within mental health care as a means of increasing access and scalability. At the same time, these systems introduce distinct safety risks arising from both user disclosures (eg, self-harm ideation) and inappropriate or inadequate AI responses. Objective: This study aimed to develop and evaluate the Automated Safety Testing and Reporting Application (ASTRA), an external system intended to identify clinically relevant risk behaviors across entire AI-mediated mental health conversations. Methods: ASTRA was tested on a dataset of 100 synthetic therapeutic conversations written by licensed clinicians to reflect risk behaviors and harmful responses between users and AI tools. Conversations varied in length and included both subtle and overt risk behavior examples across 8 predefined categories. Human coder consensus ratings served as the reference standard. ASTRA’s classifications were evaluated across 2 prompt iterations using standard diagnostic performance metrics and agreement statistics. Results: ASTRA demonstrated consistently high concordance with expert human ratings across all categories. Accuracy exceeded 0.90 for all risk behavior categories examined, with specificity uniformly high and sensitivity varying by category (range 0.55-1.00). Agreement beyond chance was substantial to almost perfect between ASTRA and human raters (κ=0.65-1.00). Detection of user self-harm indicators was particularly accurate, even in conversations where risk was expressed subtly. Conclusions: In this initial validation study, ASTRA reliably identified multiple forms of mental health–related risk behaviors at the conversation level. These findings support the feasibility of independent safety monitoring systems as a complement to AI tools used in mental health contexts and underscore the need for further evaluation using larger and real-world datasets.Keywords
All Related Versions
This publication has 27 references indexed in Scilit:
- Leveraging simulation to provide a practical framework for assessing the novel scope of risk of LLMs in healthcaremedRxiv, 2025
- ‘For Argument’s Sake, Show Me How to Harm Myself!’: Jailbreaking LLMs in Suicide and Self-Harm Contexts2025
- Performance of mental health chatbot agents in detecting and managing suicidal ideationScientific Reports, 2025
- Digital Mental Health Tools and AI Therapy Chatbots: A Balanced Approach to RegulationHastings Center Report, 2025
- Randomized Trial of a Generative AI Chatbot for Mental Health TreatmentNEJM AI, 2025
- Towards Culturally Adaptive Large Language Models in Mental Health: Using ChatGPT as a Case Study2024
- Enhancing mental health with Artificial Intelligence: Current trends and future prospectsJournal of Medicine, Surgery, and Public Health, 2024
- Chatbots and mental health: Insights into the safety of generative AIJournal of Consumer Psychology, 2023
- Using AI chatbots to provide self-help depression interventions for university students: A randomized trial of effectivenessInternet Interventions, 2022
- Barriers to healthcare access among U.S. adults with mental health challenges: A population-based studySSM - Population Health, 2021