A permutation test for assessing the presence of individual differences in treatment effects
Authors:
- Chi ChangCCChi Chang
- M. Lee Van HornMHM. Lee Van Horn
- The Pooled Resource Open-Access ALS Clinical Trials ConsortiumTCThe Pooled Resource Open-Access ALS Clinical Trials Consortium
- 27 September 2021
- journal article
- research article
- Published by SAGE Publications in Statistical Methods in Medical Research
- Vol. 30 (11) , 2369-2381
Abstract
An important goal of personalized medicine is to identify heterogeneity in treatment effects and then use that heterogeneity to target the intervention to those most likely to benefit. Heterogeneity is assessed using the predicted individual treatment effects framework, and a permutation test is proposed to establish if significant heterogeneity is present given the covariates and predictive model or algorithm used for predicted individual treatment effects. We first show evidence for heterogeneity in the effects of treatment across an illustrative example data set. We then use simulations with two different predictive methods (linear regression model and Random Forests) to show that the permutation test has adequate type-I error control. Next, we use an example dataset as the basis for simulations to demonstrate the ability of the permutation test to find heterogeneity in treatment effects for a predicted individual treatment effects estimate as a function of both effect size and sample size. We find that the proposed test has good power for detecting heterogeneity in treatment effects when the heterogeneity was due primarily to a single predictor, or when it was spread across the predictors. Power was found to be greater for predictions from a linear model than from random forests. This non-parametric permutation test can be used to test for significant differences across individuals in predicted individual treatment effects obtained with a given set of covariates using any predictive method with no additional assumptions.Keywords
All Related Versions
Funding Information
- UK Medical Research Council (# MR/L010658/1)
This publication has 37 references indexed in Scilit:
- Modeling Heterogeneous Treatment Effects in Survey Experiments with Bayesian Additive Regression TreesPublic Opinion Quarterly, 2012
- Subgroup identification based on differential effect search—A recursive partitioning method for establishing response to treatment in patient subpopulationsStatistics in Medicine, 2011
- Treatment Response Trajectories and Their Antecedents in Recent-Onset PsychosisJournal of Clinical Psychopharmacology, 2010
- Estimating drug effects in the presence of placebo response: Causal inference using growth mixture modelingStatistics in Medicine, 2009
- Treatment Fidelity in Multisite EvaluationAmerican Journal of Evaluation, 2009
- Nonparametric Tests for Treatment Effect HeterogeneityReview of Economics and Statistics, 2008
- Modelling treatment‐effect heterogeneity in randomized controlled trials of complex interventions (psychological treatments)Statistics in Medicine, 2007
- Causal Inference Using Potential OutcomesJournal of the American Statistical Association, 2005
- Neural networks for classification: a surveyIEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews), 2000
- Conditional Permutation Tests and the Propensity Score in Observational StudiesJournal of the American Statistical Association, 1984