Development, Feasibility, Acceptability, and Usability of an Artificial Intelligence–Powered Chatbot (Suzy) to Support Patients in Substance Use Disorder Recovery: Multiphase Study
Authors:
- Warren Scott ComuladaWCWarren Scott Comulada
- Dallas SwendemanDSDallas Swendeman
- Y. Xian HoYHY. Xian Ho
- Joanna M StreckJSJoanna M Streck
- Delta-Marie LewisDLDelta-Marie Lewis
- Roxana RezaiRRRoxana Rezai
- David WarrenDWDavid Warren
- Gladys PachasGPGladys Pachas
- Maria ChandlerMCMaria Chandler
- Jonathan L JacksonJJJonathan L Jackson
- Lillian GelbergLGLillian Gelberg
Open Access
- 20 May 2026
- journal article
- Published by JMIR Publications Inc. in JMIR Formative Research
- Vol. 10, e84683
Abstract
Background: Substance use disorder (SUD) remains a major public health crisis in the United States, with significant challenges in treatment access, retention, and workforce capacity. SUD care teams, including addiction medicine physicians and peer recovery coaches (PRCs), support patients receiving SUD treatment but face heavy workloads and burnout. Artificial intelligence (AI) innovations, particularly large language model (LLM)–based chatbots, may extend PRC support and provide patients with on-demand recovery support between clinic visits and PRC contacts. However, evidence on their development, feasibility, acceptability, and usability in addiction services remains limited. Objective: This study describes the development, feasibility, acceptability, and usability of an AI-powered health coaching chatbot (Suzy) designed to support patients in SUD recovery. Methods: A total of 2 clinicians, 5 researchers, and 2 technology developers led a small, multiphase pilot study. In the formative phase, they conducted focus groups and qualitative in-depth interviews with 12 health care professionals and 8 patients with substance use histories to specify chatbot functions and develop a rule-based chatbot. In phase 2, they conducted usability testing of the rule-based chatbot with 8 patients who reported substance use and completed standardized tasks, surveys, and qualitative interviews. Measures included the System Usability Scale (SUS), Net Promoter Score (NPS), and Single Ease of Use Question (SEQ). In phase 3, they developed an LLM-based chatbot co-designed and fine-tuned with PRCs and other SUD experts. Results: Rule-based chatbot functions included craving management, appointment reminders, resource referrals, care team contacts, and goal setting. Usability task testing supported feasibility. In this small pilot sample, quantitative and qualitative feedback indicated acceptability and usability, with an average SUS score of 93 (benchmark 68), an NPS of 63 (benchmark 35), and a mean SEQ score of 6.5/7. Patients valued Suzy’s approachable, nonjudgmental language and features that promoted accountability, self-monitoring, and 24/7 availability, while emphasizing that chatbots should supplement but not replace human support. The LLM-based chatbot development emphasized information accuracy, safety escalation protocols to mitigate risks of inappropriate chatbot responses, human-in-the-loop features, and expanded conversational flexibility and personal tailoring. Conclusions: In this pilot study, a rule-based chatbot designed to support SUD care demonstrated feasibility, usability, and acceptability. LLM-based chatbot development required more robust safety and emergency reporting features, while offering more patient-responsive conversational functions. By providing on-demand coaching, referrals, and reminders, Suzy may extend the reach of care teams, alleviate provider burden, and enhance patient engagement. Additional work is needed to understand how to best integrate Suzy into patients’ recovery journeys to ensure human support remains accessible and prioritized. LLM evaluation was based on expert testing and safety review. Clinical effectiveness, including the impact on substance use, was not evaluated. Next steps include evaluating the LLM chatbot in real-world settings with larger samples and assessing its efficacy in reducing substance use.Keywords
All Related Versions
This publication has 58 references indexed in Scilit:
- Collaborative Care Model for Patients With Opioid Use Disorder and Mental IllnessJAMA Network Open, 2024
- Lessons learned from employing Medicaid-funded peer recovery support specialists in residential substance use treatment settings: An exploratory analysisJournal of Substance Use and Addiction Treatment, 2023
- Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media ForumJAMA Internal Medicine, 2023
- Frequency of reported methamphetamine use linked to prevalence of clinical conditions, sexual risk behaviors, and social adversity in diverse men who have sex with men in Los AngelesDrug and Alcohol Dependence, 2022
- A randomized controlled trial of a therapeutic relational agent for reducing substance misuse during the COVID-19 pandemicDrug and Alcohol Dependence, 2021
- Enacting ‘more-than-human’ care: Clients’ and counsellors’ views on the multiple affordances of chatbots in alcohol and other drug counsellingInternational Journal of Drug Policy, 2021
- Assessment of Annual Cost of Substance Use Disorder in US HospitalsJAMA Network Open, 2021
- Comparative Effectiveness of Different Treatment Pathways for Opioid Use DisorderJAMA Network Open, 2020
- The Recovery Agenda: The Shared Role of Peers and ProfessionalsPublic Health Reviews, 2013
- Can Substance Use Disorders be Managed Using the Chronic Care Model? Review and Recommendations from a NIDA Consensus GroupPublic Health Reviews, 2013