An LLM-based Motivational Interviewing Conversational Agent for Health Behaviour Change: Comparative Evaluation Study (Preprint)
Preprint
- 26 February 2026
- preprint
- Published by JMIR Publications Inc. in JMIR Preprints
Abstract
BACKGROUND Motivational interviewing (MI) is an effective approach for supporting health behaviour change, but face-to-face delivery is resource-intensive and difficult to scale. Rule-based Conversational Agents (CA) can improve access, yet their scripted interactions and limited language flexibility constrain MI delivery. This study evaluates the feasibility of using Large Language Models (LLMs) to deliver text-based MI coaching sessions. OBJECTIVE To describe the development of an LLM-based MI CA “Aimi”, and to evaluate its performance alongside a human coach delivering text-based MI and a rule-based MI-inspired CA in a physical activity behaviour change context. METHODS We developed Aimi using structured LLM-workflows that aim to enhance MI fidelity. We conducted a within-subjects study, where 18 adults interacted with (i) Aimi, (ii) an MI-trained human coach and (iii) a rule-based CA during live text-based role-play coaching sessions, in a randomised order. Transcripts were independently evaluated by an MI expert using the Manual for the Motivational Interviewing Skill Code Version 2.0 (MISC-2) to assess MI competency and fidelity. Participants completed a user experience questionnaire to provide general feedback and to assess session alliance, dialogue relevance, empathy, engagement, linguistic quality, and perceived motivation to change. Qualitative feedback was thematically summarized and categorized under strengths and weaknesses for each approach. RESULTS Across MISC-2 summary metrics, Aimi achieved higher fidelity scores than the trained human coach and rule-based CA, showing higher reflection-to-question ratios, more complex reflections, and greater elicitation of client change talk (91% vs. 71%). User experience ratings showed no significant differences across conditions. Qualitative analysis revealed distinct strengths and limitations across the coaching interactions: participants describing Aimi’s interactions as personalized, fluid, and adaptive, though sometimes overly reflective and lengthy; the human coach was viewed as empathetic and supportive but slow to respond; and the rule-based coach was viewed as efficient and structured, yet limited in depth and personalisation. CONCLUSIONS This study demonstrates that LLM-based CAs can deliver MI with fidelity comparable to trained coaches while more reliably eliciting client change talk than trained humans or rule-based systems. The perceived robotic quality and response style represent important areas for future refinement. LLM-based CAs orchestrated through structured workflows offer a scalable pathway to deliver MI for health behaviour change.Keywords
This publication has 50 references indexed in Scilit:
- LLM-based conversational agents for behaviour change support: A randomised controlled trial examining efficacy, safety, and the role of user behaviourInternational Journal of Human-Computer Studies, 2025
- Development and preliminary testing of a secure large language model-based chatbot for brief alcohol counseling in young adultsDrug and Alcohol Dependence, 2025
- Designing a Large Language Model-Based Coaching Intervention for Lifestyle Behavior Change2024
- Global burden and strength of evidence for 88 risk factors in 204 countries and 811 subnational locations, 1990–2021: a systematic analysis for the Global Burden of Disease Study 2021The Lancet, 2024
- Ethical and regulatory challenges of large language models in medicineThe Lancet Digital Health, 2024
- Changing Behavior Using the Transtheoretical Model2020
- Effect of lifestyle interventions on glucose regulation among adults without impaired glucose tolerance or diabetes: A systematic review and meta-analysisDiabetes Research and Clinical Practice, 2017
- Development and validation of a 6-item working alliance questionnaire for repeated administrations during psychotherapy.Psychological Assessment, 2015
- A reusable framework for health counseling dialogue systems based on a behavioral medicine ontologyJournal of Biomedical Informatics, 2011
- The Motivational Interviewing Skill Code: Reliability and a Critical AppraisalBehavioural and Cognitive Psychotherapy, 2005