Application of a nonrandomized stepped wedge design to evaluate an evidence-based quality improvement intervention: a proof of concept using simulated data on patient-centered medical homes
Open Access
- 21 October 2016
- journal article
- research article
- Published by Springer Nature in BMC Medical Research Methodology
- Vol. 16 (1) , 1-11
Abstract
Stepped wedge designs have gained recognition as a method for rigorously assessing implementation of evidence-based quality improvement interventions (QIIs) across multiple healthcare sites. In theory, this design uses random assignment of sites to successive QII implementation start dates based on a timeline determined by evaluators. However, in practice, QII timing is often controlled more by site readiness. We propose an alternate version of the stepped wedge design that does not assume the randomized timing of implementation while retaining the method’s analytic advantages and applying to a broader set of evaluations. To test the feasibility of a nonrandomized stepped wedge design, we developed simulated data on patient care experiences and on QII implementation that had the structures and features of the expected data from a planned QII. We then applied the design in anticipation of performing an actual QII evaluation. We used simulated data on 108,000 patients to model nonrandomized stepped wedge results from QII implementation across nine primary care sites over 12 quarters. The outcome we simulated was change in a single self-administered question on access to care used by Veterans Health Administration (VA), based in the United States, as part of its quarterly patient ratings of quality of care. Our main predictors were QII exposure and time. Based on study hypotheses, we assigned values of 4 to 11 % for improvement in access when sites were first exposed to implementation and 1 to 3 % improvement in each ensuing time period thereafter when sites continued with implementation. We included site-level (practice size) and respondent-level (gender, race/ethnicity) characteristics that might account for nonrandomized timing in site implementation of the QII. We analyzed the resulting data as a repeated cross-sectional model using HLM 7 with a three-level hierarchical data structure and an ordinal outcome. Levels in the data structure included patient ratings, timing of adoption of the QII, and primary care site. We were able to demonstrate a statistically significant improvement in adoption of the QII, as postulated in our simulation. The linear time trend while sites were in the control state was not significant, also as expected in the real life scenario of the example QII. We concluded that the nonrandomized stepped wedge design was feasible within the parameters of our planned QII with its data structure and content. Our statistical approach may be applicable to similar evaluations.Keywords
Funding Information
- Veterans Affairs (XVA 65 - 018, IIR 11 - 326)
This publication has 16 references indexed in Scilit:
- Impact of Medical Home Implementation Through Evidence-based Quality Improvement on Utilization and CostsMedical Care, 2016
- Logistic, ethical, and political dimensions of stepped wedge trials: critical review and case studiesTrials, 2015
- Stepped wedge designs could reduce the required sample size in cluster randomized trialsJournal of Clinical Epidemiology, 2013
- Systematic review of stepped wedge cluster randomized trials shows that design is particularly used to evaluate interventions during routine implementationJournal of Clinical Epidemiology, 2011
- Statistical Methods for Quality Improvement2011
- Intent-to-treat vs. Non-intent-to-treat Analyses under Treatment Non-adherence in Mental Health Randomized TrialsPsychiatric Annals, 2008
- Alternatives to the Randomized Controlled TrialAmerican Journal of Public Health, 2008
- Design and analysis of stepped wedge cluster randomized trialsContemporary Clinical Trials, 2007
- The stepped wedge trial design: a systematic reviewBMC Medical Research Methodology, 2006
- Semiparametric Difference-in-Differences EstimatorsReview of Economic Studies, 2005