The normality assumption on between-study random effects was questionable in a considerable number of Cochrane meta-analyses
Authors:
Open Access
- 29 March 2023
- journal article
- research article
- Published by Springer Nature in BMC Medicine
- Vol. 21 (1) , 1-12
Abstract
Background: Studies included in a meta-analysis are often heterogeneous. The traditional random-effects models assume their true effects to follow a normal distribution, while it is unclear if this critical assumption is practical. Violations of this between-study normality assumption could lead to problematic meta-analytical conclusions. We aimed to empirically examine if this assumption is valid in published meta-analyses. Methods: In this cross-sectional study, we collected meta-analyses available in the Cochrane Library with at least 10 studies and with between-study variance estimates > 0. For each extracted meta-analysis, we performed the Shapiro–Wilk (SW) test to quantitatively assess the between-study normality assumption. For binary outcomes, we assessed between-study normality for odds ratios (ORs), relative risks (RRs), and risk differences (RDs). Subgroup analyses based on sample sizes and event rates were used to rule out the potential confounders. In addition, we obtained the quantile–quantile (Q–Q) plot of study-specific standardized residuals for visually assessing between-study normality. Results: Based on 4234 eligible meta-analyses with binary outcomes and 3433 with non-binary outcomes, the proportion of meta-analyses that had statistically significant non-normality varied from 15.1 to 26.2%. RDs and non-binary outcomes led to more frequent non-normality issues than ORs and RRs. For binary outcomes, the between-study non-normality was more frequently found in meta-analyses with larger sample sizes and event rates away from 0 and 100%. The agreements of assessing the normality between two independent researchers based on Q–Q plots were fair or moderate. Conclusions: The between-study normality assumption is commonly violated in Cochrane meta-analyses. This assumption should be routinely assessed when performing a meta-analysis. When it may not hold, alternative meta-analysis methods that do not make this assumption should be considered.Keywords
Funding Information
- National Institute of Mental Health (R03 MH128727)
- U.S. National Library of Medicine (R01 LM012982)
This publication has 58 references indexed in Scilit:
- Multivariate meta‐analysis: Potential and promiseStatistics in Medicine, 2011
- Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: A simulation studyStatistical Methods in Medical Research, 2010
- Confidence intervals for random effects meta-analysis and robustness to publication biasStatistics in Medicine, 2010
- Exact and efficient inference procedure for meta-analysis and its application to the analysis of independent 2 x 2 tables with all available data but without artificial continuity correctionBiostatistics, 2008
- Meta‐analysis of skewed data: Combining results reported on log‐transformed or raw scalesStatistics in Medicine, 2008
- Comparison of Two Methods to Detect Publication Bias in Meta-analysisJAMA, 2006
- A Bayesian Semiparametric Model for Random-Effects Meta-AnalysisJournal of the American Statistical Association, 2005
- Issues in the selection of a summary statistic for meta‐analysis of clinical trials with binary outcomesStatistics in Medicine, 2002
- Meta-analysis in clinical trialsControlled Clinical Trials, 1986
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960