Robbie C. M. van Aert

Univariate meta-analysis models assume that all effect sizes are independent. This assumption is violated if, for example, two outcomes are reported in a study of interest to the meta-analyst or a study reports multiple experiments administered by the same researchers. The multivariate and multilevel meta-analysis models allow modeling dependent effect sizes. These two models have recently gained in popularity among meta-analysts in psychology, where one of the largest threats to validity of meta-analysis is publication bias where a selective subset of studies remain unpublished. We extend the univariate step function selection model approach, for modeling publication bias, to multivariate and multilevel meta-analyses. We propose strict and relaxed selection models that assign a different publication probability to studies that have only statistically significant outcomes or at least one significant outcome. These two selection models are straightforward approaches for modeling publication bias. Our method can also accommodate more sophisticated alternative selection models, and we discuss possibilities for this. We primarily focus on the method's implementation as a sensitivity analysis where publication probabilities in the selection model are assumed as known but also show how these can be estimated. Simulation studies tailored to two applications show that the proposed method as a sensitivity analysis outperforms the multivariate and multilevel meta-analysis models that do not correct for publication bias. The simulations also show that the publication probability is imprecisely estimated. We offer guidance for applying the proposed method in practice and share R code to facilitate the application of our proposed methodology. (PsycInfo Database Record (c) 2026 APA, all rights reserved).

Replications are more and more conducted in multiple research fields. A recent large-scale replication project, Systematizing Confidence in Open Research and Evidence (SCORE) project, replicated 274 effects from research on business, economics, education, political science, psychology, and sociology. An open question remains how replication success can best be assessed. Thirteen methods for assessing replication success were applied in the SCORE project. The first goal of this paper is studying the statistical properties of replication success methods by a numerical study tailored to the characteristics of the SCORE data. The results of the numerical study show that replication success methods are unable to reliably draw a conclusion about replication success for typical sample sizes of the original study and replication. Hence, there is often insufficient information in one original study and one replication for drawing definite conclusions. The second goal of this paper is taking a novel perspective on the SCORE data by estimating the effect size of an original study and replication. Estimators based on meta-analysis methods that correct for potential bias in the original study were close to unbiased when synthesizing the original study and replication. The reanalysis of the SCORE data revealed that meta-analytic estimates were substantially closer to the effect size of the replication compared to the original study. The null hypothesis of no effect was rejected by the meta-analysis methods in 59.2% to 81.3% of the pairs of an original study and replication. We recommend researchers to make sure that methods for analyzing replicability have favorable statistical properties for the data at hand before applying these methods.

Data exclusion is common in psychology and can potentially impact statistical results and conclusions when applied as a p-hacking strategy. We studied the impact of data exclusion on p values in the framework of Null Hypothesis Significance Testing and developed a simulation-based methodology, along with a Shiny app, to assess the robustness of statistically significant results under different mechanisms of data exclusion. Additionally, we applied our method to 77 t tests in a sample of 16 articles published in the Journal of Personality and Social Psychology in 2014. We found that statistical conclusions (i.e., rejection of H0 given α = .05) are robust to randomized data exclusion (RDE) if the reported two-sided p value ≤ .01, and to selecting the best of ten subsets after selective randomized data exclusion (SRDE) if the reported two-sided p value ≤ .001. A small portion of articles reported sample sizes per group before (1.3%) and after data exclusion (2.6%) among 76 independent samples t tests, complicating direct robustness analyses. Of the statistically significant t test results reported in the articles, 71% were robust to RDE and 56% were robust to SRDE. Besides detailed study preregistration to prevent potential bias from data exclusion, we recommend that authors explicitly report the reasons for exclusion and provide detailed information about the sample sizes before and after data exclusion.

Crowd-sourced replication initiatives reveal that cultural differences in findings, while undoubtedly important in some cases, are often smaller and less systematic than assumed. This makes empirical testing, rather than presumption, exigent. The extent to which developmental psychology theories generalize across cultures is an empirical question, best answered via small- and large-scale collaborations between partner laboratories across the world.

Pursuing replicability — independent evidence for previous claims — is important for creating generalizable knowledge^1,2. Here we attempted replications of 274 claims of positive results from 164 quantitative papers published from 2009 to 2018 in 54 journals in the social and behavioural sciences. Replications were high powered on average to detect the original effect size (median of 99.6%), used original materials when relevant and available, and were peer reviewed in advance through a standardized internal protocol. Replications showed statistically significant results in the original pattern for 151 of 274 claims (55.1% (95% confidence interval (CI) 49.2–60.9%)) and for 80.8 of 164 papers (49.3% (95% CI 43.8–54.7%)), weighed for replicating multiple claims per paper. We observed modest variation in replication rates across disciplines (42.5–63.1%), although some estimates had high uncertainty. The median Pearson’s r effect size was 0.25 (95% CI 0.21–0.27) for original studies and 0.10 (95% CI 0.09–0.13) for replication studies, an 82.4% (95% CI 67.8–88.2%) reduction in shared variance. Thirteen methods for evaluating replication success provided estimates ranging from 28.6% to 74.8% (median of 49.3%). Some decline in effect size and significance is expected based on power to detect original effects and regression to the mean because we replicated only positive results. We observe that challenges for replicability extend across social–behavioural sciences, illustrating the importance of identifying conditions that promote or inhibit replicability^3,4.

Publication bias is a major threat to the validity of a meta-analysis, resulting in overestimated effect sizes. We propose a generalization and improvement of the publication bias method p- uniform called p -uniform*. P -uniform* improves upon p -uniform in three ways, as it (i) entails a more efficient estimator, (ii) eliminates the overestimation of effect size caused by between-study variance in true effect sizes, and (iii) enables estimating and testing for the presence of the between-study variance. We compared the statistical properties of p -uniform* with p- uniform, two implementations of the three-parameter selection model (3PSM) approach, and the random-effects model. Statistical properties of p -uniform* and 3PSM were comparable and generally outperformed p- uniform and the random-effects model if publication bias was present. We explain that p- uniform* uses a more parsimonious model than 3PSM and demonstrate that both methods estimate average effect size and between-study variance rather well with ten or more studies in the meta-analysis when publication bias is not extreme. We re-analyze the data of two published meta-analyses using p -uniform, p- uniform*, and 3PSM to illustrate the impact of publication bias on the results. We also offer recommendations for applied researchers, and we share R code in an R package as well as an easy-to-use web application for applying p -uniform*.

Bayesian hypothesis testing via Bayes factors offers a principled alternative to classical p -value methods in meta-analysis, particularly suited to its cumulative and sequential nature. Unlike p -values, Bayes factors allow for quantifying support both for and against the existence of an effect, facilitate ongoing evidence monitoring, and maintain coherent long-run behavior as additional studies are incorporated. Recent theoretical developments further show how Bayes factors can flexibly control Type I error rates through connections to e-value theory. Despite these advantages, their use remains limited in the meta-analytic literature. This article provides a critical overview of their theoretical properties, methodological considerations—such as prior sensitivity—and practical advantages for evidence synthesis. Two illustrative applications are provided: one on statistical learning in individuals with language impairments, and another on seroma incidence following post-operative exercise in breast cancer patients. New tools supporting these methods are available in the open-source R package BFpack.

Preregistration is gaining ground in psychology, and a consequence of this is that preregistered studies are more often included in meta-analyses. Preregistered studies likely mitigate the effect of publication bias in a meta-analysis, because preregistered studies can be located in the registries they were registered in even if they do not get published. However, current meta-analysis methods do not take into account that preregistered studies are less susceptible to publication bias. Traditional methods treat all studies as equivalent while meta-analytic conclusions can be improved by taking advantage of preregistered studies. The goal of this article is to introduce the hybrid extended meta-analysis (HYEMA) method that takes into account whether a study is preregistered or not and corrects for publication bias in only the nonpreregistered studies. The proposed method is applied to two meta-analyses on prominent effects in the psychological literature: the red-romance hypothesis and money priming. Applying HYEMA to these meta-analyses shows that the average effect size estimate is substantially closer to zero than the estimate of the random-effects meta-analysis model. Two simulation studies tailored to the two applications are also presented to illustrate the method's superior performance compared to the random-effects meta-analysis model and precision-effect test and precision-effect estimate with standard error when publication bias is present. Hence, I recommend to apply HYEMA as a sensitivity analysis if a mix of both preregistered and nonpreregistered studies are present in a meta-analysis. R code as well as a web application (https://rcmvanaert.shinyapps.io/HYEMA) have been developed and are described in the article to facilitate application of the method. (PsycInfo Database Record (c) 2025 APA, all rights reserved).

This research provides a comprehensive, quantitative synthesis of the empirical literature on social capital and economic growth. It assesses 957 estimates from 83 studies. While our preferred estimate of the overall mean effect is close to zero and statistically insignificant, the main finding of our analysis is the substantial degree of effect heterogeneity. We find that social capital effects can range from large negative to large positive, suggesting that its impact on economic growth varies substantially depending on the context. However, our investigation was unable to trace the sources of this heterogeneity to any observable data, estimation, and study characteristics. This suggests that other factors, not included in our study, are responsible. Our analysis did uncover significant publication bias, indicating that the estimates of social capital's effects in the empirical literature are overstated. A further finding from our analysis is that we found no evidence that different types of social capital have different effects on economic growth.

A wide variety of methods are available to estimate the between-study variance under the univariate random-effects model for meta-analysis. Some, but not all, of these estimators have been extended so that they can be used in the multivariate setting. We begin by extending the univariate generalised method of moments, which immediately provides a wider class of multivariate methods than was previously available. However, our main proposal is to use this new type of estimator to derive multivariate multistep estimators of the between-study covariance matrix. We then use the connection between the univariate multistep and Paule-Mandel estimators to motivate taking the limit, where the number of steps tends toward infinity. We illustrate our methodology using two contrasting examples and investigate its properties in a simulation study. We conclude that the proposed methodology is a fully viable alternative to existing estimation methods, is well suited to sensitivity analyses that explore the use of alternative estimators, and should be used instead of the existing DerSimonian and Laird-type moments based estimator in application areas where data are expected to be heterogeneous. However, multistep estimators do not seem to outperform the existing estimators when the data are more homogeneous. Advantages of the new multivariate multistep estimator include its semi-parametric nature and that it is computationally feasible in high dimensions. Our proposed estimation methods are also applicable for multivariate random-effects meta-regression, where study-level covariates are included in the model.