Testing Three Alternate Methods to Direct Observation in Measuring Use of Discrete Youth Cognitive Behavioral Techniques: A Secondary Analysis
Open Access
- 1 January 2025
- journal article
- Published by SAGE Publications in Implementation Research and Practice
- Vol. 6
Abstract
Background: Cognitive behavioral therapy (CBT), an umbrella term for therapeutic techniques guided by cognitive behavioral theory, is an evidence-based approach for many psychiatric conditions in youth. A stronger dose of CBT delivery is thought to improve youth clinical outcomes. While a critical indicator of care quality, measuring the use of CBT techniques feasibly and affordably is challenging. Certain CBT techniques (e.g., more concrete and observable) may be easier to measure than others using low-cost methods, such as clinician self-report; however, this has not been studied. Method: To assess the concordance of three methods of measuring CBT technique use with direct observation (DO), clinicians from 27 community agencies ( n = 126; M age = 37.7 years, SD = 12.8; 76% female) were randomized 1:1:1 to a self-report, chart-stimulated recall (CSR; semistructured interviews with the chart available), or behavioral rehearsal (BR; simulated role-plays) condition. In previous work using a global score aggregating 12 CBT techniques, only BR produced scores that did not differ from DO. This secondary analysis examined the concordance of these alternate methods with DO for each discrete CBT technique, testing for differential concordance across cognitive techniques (e.g., cognitive education) compared to behavioral techniques (e.g., behavioral activation). Results: Results of three-level mixed effects regression models indicated that BR scores did not differ significantly from DO for any techniques, and for nine techniques, neither did CSR (all p s > .05). Contrastingly, self-report scores differed from DO for all but one technique, with greater concordance for behavioral than cognitive techniques ( z = −3.29, p < .001). Conclusions: Unlike previous findings using an aggregate score, we found that both BR and CSR did not differ significantly from DO for most techniques tested. These findings have implications within implementation research and usual care settings; they support multiple viable measurement methods that are less resource-intensive than DO.Keywords
Funding Information
- National Institute of Mental Health (R01 MH108551)
This publication has 57 references indexed in Scilit:
- Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems.Journal of Consulting and Clinical Psychology, 2008
- A conceptual framework for implementation fidelityImplementation Science, 2007
- Are Physicians Discussing Prostate Cancer Screening with Their Patients and Why or Why Not? A Pilot StudyJournal of General Internal Medicine, 2007
- Assessment in Medical EducationNew England Journal of Medicine, 2007
- Determinants of Multiple Informant Agreement on Child and Adolescent BehaviorJournal of Abnormal Child Psychology, 2006
- Identifying and Selecting the Common Elements of Evidence Based Interventions: A Distillation and Matching ModelMental Health Services Research, 2005
- Informant Discrepancies in the Assessment of Childhood Psychopathology: A Critical Review, Theoretical Framework, and Recommendations for Further Study.Psychological Bulletin, 2005
- A Symptom-Level Examination of Parent–Child Agreement in the Diagnosis of Anxious YouthsJournal of the American Academy of Child & Adolescent Psychiatry, 2004
- Development of the Therapy Procedures Checklist: A Therapist-Report Measure of Technique Use in Child and Adolescent TreatmentJournal of Clinical Child & Adolescent Psychology, 2002
- Mechanisms of change in multisystemic therapy: Reducing delinquent behavior through therapist adherence and improved family and peer functioning.Journal of Consulting and Clinical Psychology, 2000