A Chatbot for the Management of Bipolar Disorder: Using Retrieval-Augmented Generation With an Open-Weight Large Language Model to Answer Clinical Questions Based on the CANMAT and ISBD 2018 Guidelines for Bipolar Disorder: Un dialogueur pour la prise en charge du trouble bipolaire : utiliser la génération augmentée par récupération avec un grand modèle de langage (GML) à poids ouverts pour répondre aux questions cliniques fondées sur les lignes directrices de 2018 de CANMAT et de l'ISBD relatives au trouble bipolaire
Open Access
- 11 June 2026
- journal article
- research article
- Published by SAGE Publications in The Canadian Journal of Psychiatry
Abstract
Objective: Clinical practice guidelines support evidence-based care but are often underused due to complexity, time constraints, and navigation challenges. We investigated whether a conversational agent (chatbot) using an open-weight large language model (LLM) with retrieval-augmented generation (RAG) could provide guideline-consistent answers for bipolar disorder management based on the full 2018 Canadian Network for Mood and Anxiety Treatments (CANMAT) and ISBD guidelines, comparing against a system using only the base LLM. Method: We developed a multi-step RAG-based chatbot that retrieves relevant guideline sections and generates responses using Llama 3.3 70B. Twenty-one clinical vignettes spanning all guideline sections were created. Six expert psychiatrists generated queries and were presented with paired responses without labels from 2 systems: one using the base Llama 3.3 70B model, the other RAG-enhanced. Responses were rated for guideline consistency on a 3-point scale, and were analyzed using mixed-effects ordinal logistic regression. Results: Experts evaluated 126 responses, of which 110 (87.3%) were rated as correct as or more correct than the baseline system. The RAG system produced 80 answers (63.5%) rated fully consistent with the guidelines versus 24 (19.0%) for baseline, and only 10 answers with major deviation (7.9%) versus 48 (38.1%) for baseline. Ordinal regression showed RAG responses were significantly more likely to be more correct (OR = 9.1, 95% CI [5.3-16.3], P < 0.001), which was consistent across all raters. Preference ratings favoured RAG answers in 78.7% of cases. Performance varied by vignette, with some errors in both retrieval and reasoning. Conclusion: The use of RAG with an open-weight model helped produce answers consistent with the CANMAT guidelines across vignettes that required adapting or combining guideline text, suggesting a proof-of-concept of a bipolar guideline chatbot. We identified areas to improve results and evaluation. Future work should explore additional retrieval strategies and LLMs, and test in more naturalistic settings.Keywords
Funding Information
- Amazon Web Services
This publication has 36 references indexed in Scilit:
- Bipolar disorderThe Lancet, 2025
- Retrieval-augmented generation elevates local LLM quality in radiology contrast media consultationnpj Digital Medicine, 2025
- Can Open-Source AI Models Diagnose Complex Cases as Well as GPT-4?JAMA, 2025
- Evaluating base and retrieval augmented LLMs with document or online support for evidence based neurologynpj Digital Medicine, 2025
- Evaluation of Retrieval-Augmented Generation: A Survey2025
- Custom Large Language Models Improve Accuracy: Comparing Retrieval Augmented Generation and Artificial Intelligence Agents to Noncustom Models for Evidence‐Based MedicineArthroscopy, 2024
- From text to treatment: the crucial role of validation for generative large language models in health careThe Lancet Digital Health, 2024
- Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based frameworknpj Digital Medicine, 2024
- Clinical decision support for bipolar depression using large language modelsNeuropsychopharmacology, 2024
- Clinical practice guidelines to inform evidence-based clinical practiceWorld Journal of Urology, 2011