An Approach to Assess Generalizability in Comparative Effectiveness Research
Open Access
- 18 May 2015
- journal article
- research article
- Published by SAGE Publications in Medical Decision Making
- Vol. 35 (8) , 1023-1036
Abstract
Background. Policy makers require estimates of comparative effectiveness that apply to the population of interest, but there has been little research on quantitative approaches to assess and extend the generalizability of randomized controlled trial (RCT)–based evaluations. We illustrate an approach using observational data. Methods. Our example is the Whole Systems Demonstrator (WSD) trial, in which 3230 adults with chronic conditions were assigned to receive telehealth or usual care. First, we used novel placebo tests to assess whether outcomes were similar between the RCT control group and a matched subset of nonparticipants who received usual care. We matched on 65 baseline variables obtained from the electronic medical record. Second, we conducted sensitivity analysis to consider whether the estimates of treatment effectiveness were robust to alternative assumptions about whether “usual care” is defined by the RCT control group or nonparticipants. Thus, we provided alternative estimates of comparative effectiveness by contrasting the outcomes of the RCT telehealth group and matched nonparticipants. Results. For some endpoints, such as the number of outpatient attendances, the placebo tests passed, and the effectiveness estimates were robust to the choice of comparison group. However, for other endpoints, such as emergency admissions, the placebo tests failed and the estimates of treatment effect differed markedly according to whether telehealth patients were compared with RCT controls or matched nonparticipants. Conclusions. The proposed placebo tests indicate those cases when estimates from RCTs do not generalize to routine clinical practice and motivate complementary estimates of comparative effectiveness that use observational data. Future RCTs are recommended to incorporate these placebo tests and the accompanying sensitivity analyses to enhance their relevance to policy making.Keywords
This publication has 56 references indexed in Scilit:
- Effect of telehealth on use of secondary care and mortality: findings from the Whole System Demonstrator cluster randomised trialBMJ, 2012
- A Randomized Controlled Trial of Telemonitoring in Older Adults With Multiple Health Issues to Prevent Hospitalizations and Emergency Department VisitsArchives of internal medicine (1960), 2012
- Telemonitoring in Patients with Heart FailureNew England Journal of Medicine, 2010
- Generalizing Evidence From Randomized Clinical Trials to Target Populations: The ACTG 320 TrialAmerican Journal of Epidemiology, 2010
- Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity‐score matched samplesStatistics in Medicine, 2009
- Developing and evaluating complex interventions: the new Medical Research Council guidanceBMJ, 2008
- A critical appraisal of propensity‐score matching in the medical literature between 1996 and 2003Statistics in Medicine, 2007
- Case finding for patients at risk of readmission to hospital: development of algorithm to identify high risk patientsBMJ, 2006
- Research in Home-Care Telemedicine: Challenges in Patient RecruitmentTelemedicine and e-Health, 2004
- The central role of the propensity score in observational studies for causal effectsBiometrika, 1983