PUMP: Estimating Power, Minimum Detectable Effect Size, and Sample Size When Adjusting for Multiple Outcomes in Multi-Level Experiments
Open Access
- 1 January 2024
- journal article
- research article
- Published by Foundation for Open Access Statistic
- Vol. 108 (6) , 1-43
Abstract
For randomized controlled trials (RCTs) with a single intervention's impact being measured on multiple outcomes, researchers often apply a multiple testing procedure (such as Bonferroni or Benjamini-Hochberg) to adjust p values. Such an adjustment reduces the likelihood of spurious findings, but also changes the statistical power, sometimes substantially. A reduction in power means a reduction in the probability of detecting effects when they do exist. This consideration is frequently ignored in typical power analyses, as existing tools do not easily accommodate the use of multiple testing procedures. We introduce the PUMP (Power Under Multiplicity Project) R package as a tool for analysts to estimate statistical power, minimum detectable effect size, and sample size requirements for multi-level RCTs with multiple outcomes. PUMP uses a simulation-based approach to flexibly estimate power for a wide variety of experimental designs, number of outcomes, multiple testing procedures, and other user choices. By assuming linear mixed effects models, we can draw directly from the joint distribution of test statistics across outcomes and thus estimate power via simulation. One of PUMP's main innovations is accommodating multiple outcomes, which are accounted for in two ways. First, power estimates from PUMP properly account for the adjustment in p values from applying a multiple testing procedure. Second, when considering multiple outcomes rather than a single outcome, different definitions of statistical power emerge. PUMP allows researchers to consider a variety of definitions of power in order to choose the most appropriate types of power for the goals of their study. The package supports a variety of commonly used frequentist multi-level RCT designs and linear mixed effects models. In addition to the main functionality of estimating power, minimum detectable effect size, and sample size requirements, the package allows the user to easily explore sensitivity of these quantities to changes in underlying assumptions.Keywords
This publication has 24 references indexed in Scilit:
- Design-Based Ratio Estimators and Central Limit Theorems for Clustered, Blocked RCTsJournal of the American Statistical Association, 2021
- Understanding Statistical Power in Cluster Randomized Trials: Challenges Posed by Differences in Notation and TerminologyJournal of Research on Educational Effectiveness, 2014
- PowerUp!: A Tool for Calculating Minimum Detectable Effect Sizes and Minimum Required Sample Sizes for Experimental and Quasi-Experimental Design StudiesJournal of Research on Educational Effectiveness, 2013
- Why We (Usually) Don't Have to Worry About Multiple ComparisonsJournal of Research on Educational Effectiveness, 2012
- On power and sample size computation for multiple testing proceduresComputational Statistics & Data Analysis, 2010
- Power and sample size when multiple endpoints are consideredPharmaceutical Statistics, 2007
- Data Analysis Using Regression and Multilevel/Hierarchical Models2006
- Sample Size Calculation for Simulation-Based Multiple-Testing ProceduresJournal of Biopharmaceutical Statistics, 2005
- Resampling-based multiple testing for microarray data analysisTEST, 2003
- Multiple Comparisons among MeansJournal of the American Statistical Association, 1961