Safety Mechanisms and Risk Mitigation in Generative AI Mental Health Chatbots: A Systematic Scoping Review
Top Cited Papers
Open Access
- 20 May 2026
- journal article
- review article
- Published by MDPI AG in Healthcare
- Vol. 14 (10) , 1395
Abstract
Background: Generative AI (GenAI) mental health chatbots are increasingly being developed to help address persistent barriers to mental healthcare. Unlike earlier rule-based and retrieval-based systems, GenAI chatbots generate open-ended outputs that can be inaccurate and unsafe. Documented harms from general-purpose GenAI chatbots have highlighted the need for purpose-built interventions with dedicated safeguards, yet how safety is implemented in such interventions remains poorly understood. Methods: This scoping review followed the Joanna Briggs Institute methodology and PRISMA-ScR guidelines, with a prospectively registered and peer-reviewed protocol. A systematic search of seven academic databases and search engines including MEDLINE, Scopus, PsycINFO, ACM Digital Library, IEEE Xplore, Google Scholar and Consensus was conducted in July 2025. Two reviewers independently screened records and extracted data. Safety mechanisms and risk mitigation strategies were narratively synthesised across three pre-specified domains: technical safeguards, pre-deployment safety considerations, and delivery-phase risk mitigation strategies. Results: Twenty-one studies across 11 countries were included. Most interventions incorporated at least one technical safety mechanism, most commonly fine-tuning and prompt engineering. A smaller subset implemented layered safety architectures combining retrieval systems, content filters or risk classifiers, and rule-based algorithms. Pre-deployment safeguards included clinical expert and user co-design approaches, research ethics procedures, and data privacy measures. During intervention delivery, detailed onboarding with role clarification was common, but human oversight was limited. Crisis referral protocols varied in rigour but were mostly underdeveloped, and systematic adverse event monitoring was sparse. Documented safety failures included missed suicidal ideation and provision of inaccurate clinical information. Conclusions: GenAI chatbot interventions require a robust sociotechnical approach that integrates technical safeguards with user co-design, procedural controls, and human oversight. Future research is needed to evaluate efficacy, improve safeguards and standardise safety outcome measurement. Regulatory oversight proportional to the risks these systems carry is required to enable integration into stepped or blended mental healthcare.Keywords
All Related Versions
This publication has 54 references indexed in Scilit:
- Large language models as mental health resources: Patterns of use in the United States.Practice Innovations, 2026
- Performance of mental health chatbot agents in detecting and managing suicidal ideationScientific Reports, 2025
- VCounselor: a psychological intervention chat agent based on a knowledge-enhanced large language modelMultimedia Systems, 2024
- The EU Artificial Intelligence Act (2024): Implications for healthcareHealth Policy, 2024
- Closing the accessibility gap to mental health treatment with a personalized self-referral chatbotNature Medicine, 2024
- Chatbots and mental health: Insights into the safety of generative AIJournal of Consumer Psychology, 2023
- Chatbot-Delivered Psychotherapy for Adults With Depressive and Anxiety Symptoms: A Systematic Review and Meta-RegressionBehavior Therapy, 2022
- Global, regional, and national burden of 12 mental disorders in 204 countries and territories, 1990–2019: a systematic analysis for the Global Burden of Disease Study 2019The Lancet Psychiatry, 2022
- Estimating the true global burden of mental illnessThe Lancet Psychiatry, 2016
- Improving access to psychological treatments: Lessons from developing countriesBehaviour Research and Therapy, 2011