Development and validation of a cognitive behavioral therapy for psychosis online training with automated feedback.
Open Access
- 1 March 2025
- journal article
- research article
- Published by American Psychological Association (APA) in Psychotherapy
- Vol. 62 (1) , 1-11
Abstract
The accessibility of training and fidelity assessment is critical to implementing and sustaining empirically supported psychotherapies like cognitive behavioral therapy for psychosis (CBTp). We describe the development of an online CBTp training tool that incorporates behavioral rehearsal tasks to enable deliberate practice of cognitive and behavioral techniques for psychosis. The development process consisted of designing content, inclusive of didactics, client profiles, and learner prompts; constructing standardized performance tasks and metrics; collecting responses to learner prompts; establishing intraclass correlation (ICC) of responses among trained raters; and training a transformer-based machine learning (ML) model to meet or surpass human ICC. Authenticity ratings of each simulated client surpassed benchmarks. CBTp trainers (n = 12), clinicians (n = 78), and nonclinicians (n = 119) generated 3,958 unique verbal responses to 28 unique prompts (7 skills × 4 simulated clients), of which the coding team rated 1,961. Human ICC across all skills was high (mean ICC = 0.77). On average, there was a high correlation between ML and human ratings of fidelity (rs = .74). Similarly, the average percentage of human agreement was high at 96% (range = 87%-102%), where values greater than 100 indicate that the ML model agreed with a human rater more than two human raters agreed with each other. Results suggest that it is possible to reliably measure discrete CBTp skills in response to simulated client vignettes while capturing expected variation in skill utilization across participants. These findings pave the way for a standardized, asynchronous training that incorporates automated feedback on learners' rehearsal of CBTp skills. (PsycInfo Database Record (c) 2025 APA, all rights reserved).Keywords
Funding Information
- National Institute of Mental Health (R42MH123215))
This publication has 32 references indexed in Scilit:
- Examining implementer fidelity: conceptualising and measuring adherence and competenceJournal of Children's Services, 2011
- Toward the Effective and Efficient Measurement of Implementation FidelityAdministration and Policy in Mental Health and Mental Health Services Research, 2010
- Balancing fidelity and adaptation in the dissemination of empirically-supported treatments: The promise of transdiagnostic interventionsBehaviour Research and Therapy, 2009
- What Are the Components of CBT for Psychosis? A Delphi StudySchizophrenia Bulletin, 2009
- The making of an expert.2007
- Understanding the Common Elements of Evidence-Based PracticeJournal of the American Academy of Child & Adolescent Psychiatry, 2007
- Identifying and Selecting the Common Elements of Evidence Based Interventions: A Distillation and Matching ModelMental Health Services Research, 2005
- Assessing simulated patients in an educational setting: the MaSP (Maastricht Assessment of Simulated Patients)Medical Education, 2004
- Measurement of Fidelity in Psychiatric RehabilitationMental Health Services Research, 2000
- Intraclass correlations: Uses in assessing rater reliability.Psychological Bulletin, 1979