Therapy as an NLP Task: Comparing LLMs and Human Peers Behaviors in CBT Sessions
Open Access
- 23 September 2026
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Human-Computer Interaction
- Vol. 10 (6) , 1-34
Abstract
Large language models (LLMs) are increasingly being used as ad hoc therapists. While prior research has found that LLMs outperform human counselors in generating single-turn empathetic responses, fewer studies have compared their behaviors across multi-turn sessions. In this study, we compare the session-level behaviors of human peer counselors with those of an LLM, both trained on the same manual to deliver multi-turn, single-session Cognitive Behavioral Therapy (CBT). Our three-phase, mixed-methods study involved: (a) an 18-month ethnography of a peer support platform, where seven counselors iteratively refined CBT prompts through 110 self-counseling sessions and 60 weekly focus groups; (b) a novel session generation method that allows direct, controlled comparison of human and LLM counselors under matched conditions—client responses were drawn from publicly available human-led CBT sessions while counselor responses were generated by a CBT-prompted LLM; and (c) expert evaluations conducted by three licensed clinical psychologists. Through data triangulation, our results show a trade-off. Human peer counselors use relational techniques to interpret subtle cues, adapt CBT to users’ values and cultural contexts, and use strategies such as small talk and contextually relevant self-disclosure to build rapport and guide the session, but often at the expense of session structure and therapeutic focus. LLM counselors, on the other hand, demonstrate greater methodological adherence to CBT techniques, but struggle to sustain turn-taking, frequently fail to distinguish between clinically important and trivial content, and are more prone to lecturing and imposing solutions. LLM counselors also tend to produce “deceptive empathy”, excessively anthropomorphic responses that can inflate user expectations of genuine human care. Taken together, our findings imply that while LLMs may outperform human counselors when generating a single-turn interaction, their ability to lead multi-turn sessions is more limited, highlighting that therapy cannot be reduced to a standalone natural language processing (NLP) task. We conclude by mapping concrete design opportunities and ethical guardrails for hybrid human-AI systems, emphasizing the risks of over-attributing human relational subjectivity to current LLMs.Keywords
This publication has 85 references indexed in Scilit:
- Exploring the ‘Dark Side’ of AI‐Powered Digital Assistants: A Moderated Mediation Model of Antecedents and Outcomes of Perceived CreepinessJournal of Consumer Behaviour, 2025
- Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media ForumJAMA Internal Medicine, 2023
- The promise of machine learning in predicting treatment outcomes in psychiatryWorld Psychiatry, 2021
- Mixing qualitative methods versus methodologies: A critical reflection on communication and power in inpatient careCounselling and Psychotherapy Research, 2020
- Introducing Cognitive Analytic Therapy2020
- Digital Interventions for Mental Disorders: Key Features, Efficacy, and Potential for Artificial Intelligence Applications2019
- Do therapists' subjective variables impact on psychodynamic psychotherapy outcomes? A systematic literature reviewClinical Psychology & Psychotherapy, 2017
- Turning to Peers: Integrating Understanding of the Self, the Condition, and Others’ Experiences in Making Sense of Complex Chronic ConditionsComputer Supported Cooperative Work (CSCW), 2016
- National Estimates for Mental Health Mutual Support Groups, Self-Help Organizations, and Consumer-Operated ServicesAdministration and Policy in Mental Health and Mental Health Services Research, 2005
- Determinants of change in cognitive therapy for depressionCognitive Therapy and Research, 1990