External and Internal Validity Considerations in Youth Effectiveness Trials: Lessons Learned from the COMET Study
- 2 November 2023
- journal article
- research article
- Published by Taylor & Francis in Journal of Clinical Child & Adolescent Psychology
- Vol. 52 (6) , 735-749
Abstract
ObjectiveEffectiveness trials aim to increase the generalizability and public health impact of interventions. However, challenges associated with this design present threats to external and internal validity. This paper illustrates these challenges using data from a two-site randomized effectiveness trial, the Community Study of Outcome Monitoring for Emotional Disorders in Teens (COMET) and presents recommendations for future research.MethodCOMET was a randomized effectiveness trial conducted in 19 community mental health clinics in two states comparing three interventions: treatment as usual (TAU), TAU with measurement-based care (TAU+), and the Unified Protocol forTransdiagnostic Treatment of Emotional Disorders in Adolescents with MBC (UPA). Participants included 176 clinicians (mean age = 35.5; 85.8% cisgender female; 53.0% racially and/or ethnically minorized) and 196 adolescents (mean age = 14.7; 65.3% cisgender female; 69.4% racially and/or ethnically minorized). Analyses outlined participant flow from recruitment to study completion, described participant characteristics, and examined site differences.ResultsAnalysis of participant flow suggested that recruitment and retention of clinicians and adolescents was challenging, raising questions about whether participants were representative of participating clinics. Both the clinician and adolescent samples were racially and ethnically diverse and adolescents were low income and clinically complex. Significant site differences were observed in clinician and adolescent characteristics.ConclusionsWhile this study was successful in recruiting a diverse and historically under-represented sample, difficulties in recruitment and retention raise questions about external validity and site differences present challenges to internal validity of study findings. Suggestions for future effectiveness studies, drawing from implementation science approaches, are discussed.Keywords
Funding Information
- National Institute of Mental Health (MH106536)
This publication has 33 references indexed in Scilit:
- A Primer on Effectiveness and Efficacy TrialsClinical and Translational Gastroenterology, 2014
- Performance of Evidence-Based Youth Psychotherapies Compared With Usual Clinical CareJAMA Psychiatry, 2013
- Effectiveness-implementation Hybrid DesignsMedical Care, 2012
- The role of therapist training in the implementation of psychosocial treatments: A review and critique with recommendationsClinical Psychology Review, 2010
- Clinical characteristics of anxiety disordered youthJournal of Anxiety Disorders, 2010
- Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation scienceImplementation Science, 2009
- The Treatment for Adolescents With Depression Study (TADS): Demographic and Clinical CharacteristicsJournal of the American Academy of Child & Adolescent Psychiatry, 2005
- The Screen for Child Anxiety Related Emotional Disorders (SCARED): Scale Construction and Psychometric CharacteristicsJournal of the American Academy of Child & Adolescent Psychiatry, 1997
- Employee turnover in community mental health organization: A developmental stages studyCommunity Mental Health Journal, 1994
- A Children's Global Assessment Scale (CGAS)Archives of General Psychiatry, 1983