Gianmarco Altoè

Meta-analyses, like primary studies, are affected by researcher degrees of freedom: researchers must decide which studies to include, how to compute effect sizes, and which statistical models to use. Different reasonable choices can lead to different results, making multiverse meta-analysis a useful tool for assessing robustness across alternative specifications. However, multiverse meta-analysis also introduces a multiple-testing problem. Current inferential approaches mainly provide a global test, which can establish that at least one specification differs from the null but cannot identify which individual specifications do so while accounting for multiplicity across the full multiverse. We introduce Post-selection Inference in Multiverse Meta-Analysis (PIMMA), which allows inference on individual specifications while controlling the Family-Wise Error Rate (FWER) and accounting for their dependence. In practical terms, PIMMA shows which results remain statistically significant after accounting for multiplicity across all reasonable analytical choices. In simulations, the specification-level test adequately controlled Type I error, with power comparable to robust variance estimation in scenarios where both methods maintained appropriate Type I error control. In an application to meta-analyses of Cognitive Behavioral Therapy efficacy for adult depression, 297 of 336 unique meta-analytic analyses remained significant after step-down max-T adjustment for multiple testing, accounting for dependence among analyses. PIMMA therefore extends multiverse meta-analysis from a global assessment of whether any effect is present to specification-level inference, allowing researchers to evaluate robustness across specifications and identify which individual results remain significant under appropriate error control.

The probability of direction (𝑝𝑑) has seen rapid adoption in psychological research as an intuitive posterior summary that quantifies how strongly the posterior distribution favors one direction of an effect over the other. In practice, researchers often turn 𝑝𝑑 into a binary decision rule by applying fixed thresholds (e.g., 𝑝𝑑 > 0.975), declaring evidence for an effect when posterior mass concentrates strongly on one side. When this rule is applied across many parameters, it can substantially increase the risk of misleading conclusions. We argue that this problem stems from neglecting the prior probability that for every tested effect, the data-favored direction does not reflect a genuine effect. This quantity, often left implicit, becomes increasingly consequential as the number of tested effects grows. To address this, we propose a prior-odds adjustment to 𝑝𝑑 that requires researchers to explicitly specify the proportion of tested effects they expect to be null. The adjustment accounts for multiplicity without modifying the model used in parameter estimation, offers a transparent mechanism for incorporating priorskepticism about the existence of effects, and reduces the risk of false discoveries in multi-parameter settings. We evaluate its behavior through simulation across varying sample sizes and proportions of null effects, and provide an R package to facilitate applied use.

Gelman and Carlin introduced Type S (sign) and Type M (magnitude) errors to highlight the possibility that statistically significant results in published articles are misleading. Although these concepts have been proposed to be useful both when designing a study (prospective) and when evaluating results (retroactive), we argue that these statistics do not facilitate the proper design of studies or the meaningful interpretation of results. Type S errors are a response to the criticism of testing against a point null of exactly zero in contexts in which true zero effects are implausible. Testing against a minimum effect while controlling the Type 1 error rate provides a more coherent and practically useful alternative. Type M errors warn against effect-size inflation after selectively reporting significant results, but we argue that statistical indices such as the critical effect size or bias-adjusted effect size are preferable approaches. We do believe that Type S and Type M errors can be valuable in statistics education, in which the principles of error control are explained, and in the discussion section of studies that fail to follow good research practices. Overall, we argue that their use cases are more limited than is currently recognized and that alternative solutions deserve greater attention.

The Music Ensemble dataset is a large-scale, cross-national database that provides detailed information about the musical, cognitive, personality, and demographic profiles of young adult musicians and nonmusicians. Data were collected from 1438 participants (aged 18–30) across thirty-five research sites in Europe, North America, South America, and Australia. Participants completed an in-person, in-lab battery of objective tests, including measures of verbal, visuospatial and musical short-term memory, executive functions (updating component), nonverbal reasoning, verbal comprehension, and music perception skills. The battery also included standardized and custom self-report questionnaires assessing music sophistication, music reward, personality traits, socioeconomic status, and demographic characteristics. Music Ensemble was preregistered, and the research protocol followed a standardized procedure across sites. The dataset also includes a large subsample of musicians and nonmusicians that are pair-matched for age, gender, and education (678 pairs). It enables well-powered investigations into the relationship between musical expertise and individual differences in cognition, personality, and demographic variables. It is also suitable for training in statistical and psychometric methods.

Gelman and Carlin (2014) introduced Type S (sign) and Type M (magnitude) errors to highlight the possibility that statistically significant results in published articles are misleading. While these concepts have been proposed to be useful both when designing a study (prospective) and when evaluating results (retroactive), we argue that these statistics do not facilitate the proper design of studies, nor the meaningful interpretation of results. Type S errors are a response to the criticism of testing against a point null of exactly zero in contexts where true zero effects are implausible. Testing against a minimum-effect, while controlling the Type 1 error rate, provides a more coherent and practically useful alternative. Type M errors warn against effect size inflation after selectively reporting significant results, but we argue that statistical indices such as the critical effect size or bias adjusted effect size are preferable approaches. We do believe that Type S and M errors can be valuable in statistics education where the principles of error control are explained, and in the discussion section of studies that fail to follow good research practices. Overall, we argue their use-cases are more limited than is currently recognized, and alternative solutions deserve greater attention.

In this paper we test the content validity of 246 academic research value items in two interrelated methodological steps. First, we analyze the in-depth evaluation of two experts to assess formal aspects of our items and their relatedness to the construct. Next, we evaluate the relevance of each item based on feedback gathered from 20 experts. Based on these assessments, we review and refine the original item pool and propose a total of 97 value items spread through 10 dimensions. We relate our items to existing work on researchers’ values and provide recommendations for future measurement development. A validated measure of academic research values can assess the effectiveness of responsible research conduct courses and highlight personal differences among researchers from diverse backgrounds, as well as aid our understanding of values unique to academic careers.

Musicians are often regarded as a positive example of brain plasticity and associated cognitive benefits. This emerges when experienced musicians (e.g., musicians with more than 10 years of music training and practice) are compared with nonmusicians. A frequently observed behavioral finding is a short-term memory advantage of the former over the latter. Although available meta-analysis reported that the effect size of this advantage is medium (Hedges’s g = 0.5), no literature study was adequately powered to estimate reliably an effect of such size. This multilab study has been ideated, realized, and conducted in lab by several groups that have been working on this topic. Our ultimate goal was to provide a community-driven shared and reliable estimate of the musicians’ short-term memory advantage (if any) and set a method and a standard for future studies in neuroscience and psychology comparing musicians and nonmusicians. Thirty-three research units recruited a total of 600 experienced musicians and 600 nonmusicians, a number that is sufficiently large to estimate a small effect size (Hedges’s g = 0.3) with a high statistical power (i.e., 95%). Subsequently, we measured the difference in short-term memory for musical, verbal, and visuospatial stimuli. We also looked at cognitive, personality, and socioeconomic factors that might mediate the difference. Musicians had better short-term memory than nonmusicians for musical, verbal, and visuospatial stimuli with an effect size of, respectively, Hedges’s g s = 1.08 (95% confidence interval [CI] = [0.94, 1.22]; large), 0.16 (95% CI = [0.02 0.30]; very small), and 0.28 (95% CI = [0.15, 0.41]; small). This work sets the basis for sound research practices in studies comparing musicians and nonmusicians and contributes to the ongoing debate on the possible cognitive benefits of musical training.

The Music Ensemble dataset is a large-scale, cross-national database that provides detailed information about the musical, cognitive, personality, and demographic profiles of musicians and nonmusicians. Data were collected from 1438 participants (aged 18–30) across thirty-five research sites in Europe, North America, South America, and Australia. Participants completed an in-person, in-lab battery of objective tests, including measures of verbal, visuospatial and musical short-term memory, executive functions (updating component), nonverbal reasoning, verbal comprehension, and music perception skills. The battery also included standardized and custom self-report questionnaires assessing music sophistication, music reward, personality traits, socioeconomic status, and demographic characteristics. Music Ensemble was preregistered, and the research protocol followed a standardized procedure across sites. The dataset also includes a large subsample of musicians and nonmusicians that are pair-matched for age, gender, and education (678 pairs). It enables well-powered investigations into the relationship between musical expertise and individual differences in cognition, personality, and demographic variables. It is also suitable for training in statistical and psychometric methods.

Critical-effect-size values represent the smallest detectable effect that can reach statistical significance given a specific sample size, alpha level, and test statistic. It can be useful to calculate the critical effect size when designing a study and evaluate whether such effects are plausible. Reporting critical-effect-size values may be useful when the sample size has not been planned a priori, there is uncertainty about the expected sample size that can be collected, or researchers plan to analyze the data with a statistical hypothesis test. To assist researchers in calculating critical-effect-size values, we developed an R package that allows researchers to report critical-effect-size values for group comparisons, correlations, linear regressions, and meta-analyses. Reflecting on critical-effect-size values could benefit researchers during the planning phase of the study by helping them to understand the limitations of their research design. Critical-effect-size values are also useful when evaluating studies performed by other researchers when a priori power analyses were not performed, especially when nonsignificant results are observed.