Douglas E. Faries

The assumption of “no unmeasured confounders” is a critical but unverifiable assumption required for causal inference yet quantitative sensitivity analyses to assess robustness of real‐world evidence remains under‐utilized. The lack of use is likely in part due to complexity of implementation and often specific and restrictive data requirements for application of each method. With the advent of methods that are broadly applicable in that they do not require identification of a specific unmeasured confounder—along with publicly available code for implementation—roadblocks toward broader use of sensitivity analyses are decreasing. To spur greater application, here we offer a good practice guidance to address the potential for unmeasured confounding at both the design and analysis stages, including framing questions and an analytic toolbox for researchers. The questions at the design stage guide the researcher through steps evaluating the potential robustness of the design while encouraging gathering of additional data to reduce uncertainty due to potential confounding. At the analysis stage, the questions guide quantifying the robustness of the observed result and providing researchers with a clearer indication of the strength of their conclusions. We demonstrate the application of this guidance using simulated data based on an observational fibromyalgia study, applying multiple methods from our analytic toolbox for illustration purposes.

Propensity score matching is commonly used to draw causal inference from observational survival data. However, its asymptotic properties have yet to be established, and variance estimation is still open to debate. We derive the statistical properties of the propensity score matching estimator of the marginal causal hazard ratio based on matching with replacement and a fixed number of matches. We also propose a double‐resampling technique for variance estimation that takes into account the uncertainty due to propensity score estimation prior to matching.

The assumption of no unmeasured confounders is a critical but unverifiable assumption required for causal inference yet quantitative sensitivity analyses to assess robustness of real-world evidence remains underutilized. The lack of use is likely in part due to complexity of implementation and often specific and restrictive data requirements required for application of each method. With the advent of sensitivity analyses methods that are broadly applicable in that they do not require identification of a specific unmeasured confounder, along with publicly available code for implementation, roadblocks toward broader use are decreasing. To spur greater application, here we present a best practice guidance to address the potential for unmeasured confounding at both the design and analysis stages, including a set of framing questions and an analytic toolbox for researchers. The questions at the design stage guide the research through steps evaluating the potential robustness of the design while encouraging gathering of additional data to reduce uncertainty due to potential confounding. At the analysis stage, the questions guide researchers to quantifying the robustness of the observed result and providing researchers with a clearer indication of the robustness of their conclusions. We demonstrate the application of the guidance using simulated data based on a real-world fibromyalgia study, applying multiple methods from our analytic toolbox for illustration purposes.

The assumption of ‘no unmeasured confounders’ is a critical but unverifiable assumption required for causal inference yet quantitative sensitivity analyses to assess robustness of real-world evidence remains underutilized. The lack of use is likely in part due to complexity of implementation and often specific and restrictive data requirements required for application of each method. With the advent of sensitivity analyses methods that are broadly applicable in that they do not require identification of a specific unmeasured confounder – along with publicly available code for implementation – roadblocks toward broader use are decreasing. To spur greater application, here we present a best practice guidance to address the potential for unmeasured confounding at both the design and analysis stages, including a set of framing questions and an analytic toolbox for researchers. The questions at the design stage guide the research through steps evaluating the potential robustness of the design while encouraging gathering of additional data to reduce uncertainty due to potential confounding. At the analysis stage, the questions guide researchers to quantifying the robustness of the observed result and providing researchers with a clearer indication of the robustness of their conclusions. We demonstrate the application of the guidance using simulated data based on a real-world fibromyalgia study, applying multiple methods from our analytic toolbox for illustration purposes.

Unlike in randomized clinical trials (RCTs), confounding control is critical for estimating the causal effects from observational studies due to the lack of treatment randomization. Under the unconfoundedness assumption, matching methods are popular because they can be used to emulate an RCT that is hidden in the observational study. To ensure the key assumption hold, the effort is often made to collect a large number of possible confounders, rendering dimension reduction imperative in matching. Three matching schemes based on the propensity score (PSM), prognostic score (PGM), and double score (DSM, ie, the collection of the first two scores) have been proposed in the literature. However, a comprehensive comparison is lacking among the three matching schemes and has not made inroads into the best practices including variable selection, choice of caliper, and replacement. In this article, we explore the statistical and numerical properties of PSM, PGM, and DSM via extensive simulations. Our study supports that DSM performs favorably with, if not better than, the two single score matching in terms of bias and variance. In particular, DSM is doubly robust in the sense that the matching estimator is consistent requiring either the propensity score model or the prognostic score model is correctly specified. Variable selection on the propensity score model and matching with replacement is suggested for DSM, and we illustrate the recommendations with comprehensive simulation studies. An R package is available athttps://github.com/Yunshu7/dsmatch.

Aim: To predict optimal treatments maximizing overall survival (OS) and time to treatment discontinuation (TTD) for patients with metastatic breast cancer (MBC) using machine learning methods on electronic health records. Patients/methods: Adult females with HR+/HER2- MBC on first- or second-line systemic therapy were eligible. Random survival forest (RSF) models were used to predict optimal regimen classes for individual patients and each line of therapy based on baseline characteristics. Results: RSF models suggested greater use of CDK4 & 6 inhibitor-based therapies may maximize OS and TTD. RSF-predicted optimal treatments demonstrated longer OS and TTD compared with nonoptimal treatments across line of therapy (hazard ratios = 0.44∼0.79). Conclusion: RSF may help inform optimal treatment choices and improve outcomes for patients with HR+/HER2- MBC.

The FDA is preparing guidance about using real-world evidence (RWE) to support decisions about product effectiveness. Several ongoing efforts aim to replicate randomized clinical trial (RCT) results using RWE with the intent of identifying circumstances and methods that provide valid evidence of drug effects. Lack of agreement may not be due to faulty methods but rather to the challenges with emulating RCTs, differences in healthcare settings and patient populations, differences in effect measures and data analysis, bias, and/or the efficacy–effectiveness gap. In fact, for some decisions, RWE may lead to better understanding of how treatments work in usual care settings than a more constrained view from RCTs. Efforts to reconcile the role and opportunities for generating complementary evidence from RWE and RCTs will advance regulatory science.

Background: Rheumatoid arthritis (RA) is a condition with symptoms that vary over time. The typical 3- to 6-month interval between physician visits may lead to patients failing to recall or underreporting symptoms experienced during the interim. Wearable digital technology enables the regular passive collection of patients’ biometric and activity data. If it is shown to be strongly related to data captured by patient-reported outcome (PRO) measures, information collected passively from wearable digital technology could serve as an objective proxy or be complementary to patients’ subjective experience of RA symptoms. Objective: The goal of this study is to characterize the extent to which digital measures collected from a consumer-grade smartwatch agree with measures of RA disease activity and other PROs collected via a smartphone app. Methods: This observational study will last 6 months for each participant. We aim to recruit 250 members of the ArthritisPower registry with an RA diagnosis who will receive a smartwatch to wear for the period of the study. From the ArthritisPower mobile app on their own smartphone device, participants will be prompted to answer daily and weekly electronic PRO (ePRO) measures for the first 3 months. Results: The study was launched in December 2018 and will require up to 18 months to complete. Study results are expected to be published by the end of 2021. Conclusions: The completion of this study will provide important data regarding the following: (1) the relationship between passively collected digital measures related to activity, heart rate, and sleep collected from a smartwatch with ePROs related to pain, fatigue, physical function, and RA flare entered via smartphone app; (2) determine predictors of adherence with smartwatch and smartphone app technology; and (3) assess the effect of study-specific reminders on adherence with the smartwatch. International Registered Report Identifier (IRRID): DERR1-10.2196/14665

BACKGROUND Rheumatoid arthritis (RA) is a condition with symptoms that vary over time. The typical 3- to 6-month interval between physician visits may lead to patients failing to recall or underreporting symptoms experienced during the interim. Wearable digital technology enables the regular passive collection of patients’ biometric and activity data. If it is shown to be strongly related to data captured by patient-reported outcome (PRO) measures, information collected passively from wearable digital technology could serve as an objective proxy or be complementary to patients’ subjective experience of RA symptoms. OBJECTIVE The goal of this study is to characterize the extent to which digital measures collected from a consumer-grade smartwatch agree with measures of RA disease activity and other PROs collected via a smartphone app. METHODS This observational study will last 6 months for each participant. We aim to recruit 250 members of the ArthritisPower registry with an RA diagnosis who will receive a smartwatch to wear for the period of the study. From the ArthritisPower mobile app on their own smartphone device, participants will be prompted to answer daily and weekly electronic PRO (ePRO) measures for the first 3 months. RESULTS The study was launched in December 2018 and will require up to 18 months to complete. Study results are expected to be published by the end of 2021. CONCLUSIONS The completion of this study will provide important data regarding the following: (1) the relationship between passively collected digital measures related to activity, heart rate, and sleep collected from a smartwatch with ePROs related to pain, fatigue, physical function, and RA flare entered via smartphone app; (2) determine predictors of adherence with smartwatch and smartphone app technology; and (3) assess the effect of study-specific reminders on adherence with the smartwatch. INTERNATIONAL REGISTERED REPORT DERR1-10.2196/14665

Background: Osteoporosis is prevalent in the United States, with an increasing need for management. In this study, we evaluated the effectiveness of a private orthopaedic practice-based osteoporosis management service (OP MS) in reducing subsequent fracture risk and improving other aspects of osteoporosis management of patients who had sustained fractures. Methods: This was a retrospective cohort study using the 100% Medicare data set for Michigan residents with any vertebral; hip, pelvic or femoral; or other nonvertebral fracture during the period of April 1, 2010 to September 30, 2014. Patients who received OP MS care with a follow-up visit within 90 days of the first fracture, and those who did not seek OP MS care but had a physician visit within 90 days of the first fracture, were considered as exposed and unexposed, respectively (first follow-up visit = index date). Eligible patients with continuous enrollment in Medicare Parts A and B for the 90-day pre-index period were followed until the earliest of death, health-plan disenrollment, or study end (December 31, 2014) to evaluate rates of subsequent fracture, osteoporosis medication prescriptions filled, and bone mineral density (BMD) assessments. Health-care costs were evaluated among patients with 12 months of post-index continuous enrollment. Propensity-score matching was used to balance differences in baseline characteristics. Each exposed patient was matched to an unexposed patient within ± 0.01 units of the propensity score. After propensity-score matching, Cox regression examined the hazard ratio (HR) of clinical and economic outcomes in the exposed and unexposed cohorts. Results: Two well-matched cohorts of 1,304 patients each were produced. The exposed cohort had a longer median time to subsequent fracture (998 compared with 743 days; log-rank p = 0.001), a lower risk of subsequent fracture (HR = 0.8; 95% confidence interval [CI] = 0.7 to 0.9), and a higher likelihood of having osteoporosis medication prescriptions filled (HR = 1.7; 95% CI = 1.4 to 2.0) and BMD assessments (HR = 4.3; 95% CI = 3.7 to 5.0). The total 12-month costs ($25,306 compared with $22,896 [USD]; p = 0.082) did not differ significantly between the cohorts. Conclusions: A private orthopaedic practice-based OP MS effectively reduced subsequent fracture risk, likely through coordinated and ongoing comprehensive patient care, without a significant overall higher cost. Level of Evidence: Therapeutic Level III. See Instructions for Authors for a complete description of levels of evidence.