Toward trustworthy chatbots: a protocol for red teaming for health related conversations
Open Access
- 31 March 2026
- journal article
- research article
- Published by Springer Nature in Scientific Reports
- Vol. 16 (1)
Abstract
Health-related chatbots require safety assurance beyond factual correctness. We propose a red-teaming protocol for patient-facing AI structured around three pillars: error stratification, dual-pronged testing, and vulnerability-informed mitigation. We distinguish Document Adherence (DA) from Instruction Adherence (IA), deploying adversarial “attacks” across both single-turn and multi-turn exchanges to provoke system failures. We then applied layered mitigations informed by the vulnerabilities revealed by these attacks. We evaluate this framework on a retrieval-augmented generation (RAG) based chatbot designed to assist with health-related social needs (HRSN).The protocol identified behavioral noncompliance as the dominant risk. While robust in DA (0/60 errors), the system struggled with IA (15% error rate). Crucially, multi-turn stress tests revealed vulnerabilities hidden in single-turn checks: error rates spiked to 50% for advice queries and 40% for user distress. All high-severity failures occurred during these sustained interactions. Of our mitigations, prompt augmentation reduced total errors by 60%, while document augmentation mitigated single-turn distress errors. Combined, they eliminated high-severity errors entirely by forcing “safe failure” loops. We suggest this cycle of stratified analysis, depth-based testing, and targeted mitigation can be a guiding framework for securing clinical conversational agents.Keywords
Funding Information
- The Ohio State University Clinical and Translational Science Institute
- National Center for Advancing Translational Sciences of the National Institutes of Health (UM1TR004548)
This publication has 22 references indexed in Scilit:
- Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers.2025
- Randomized Trial of a Generative AI Chatbot for Mental Health TreatmentNEJM AI, 2025
- Red teaming ChatGPT in medicine to yield real-world insights on model behaviornpj Digital Medicine, 2025
- Medical Hallucination in Foundation Models and Their Impact on HealthcaremedRxiv, 2025
- Transforming healthcare with chatbots: Uses and applications—A scoping reviewDigital Health, 2025
- Clinical Reasoning of a Generative Artificial Intelligence Model Compared With PhysiciansJAMA Internal Medicine, 2024
- A systematic review of artificial intelligence-powered (AI-powered) chatbot intervention for managing chronic illnessAnnals of Medicine, 2024
- Lost in the Middle: How Language Models Use Long ContextsTransactions of the Association for Computational Linguistics, 2024
- Improving Social Needs Intervention Research: Key Questions for Advancing the FieldAmerican Journal of Preventive Medicine, 2019
- Inter-Coder Agreement for Computational LinguisticsComputational Linguistics, 2008