James G. Field

Published claims should be reproducible, yielding the same result when applying the same analysis to the same data. We assessed reproducibility in a stratified random sample of 600 papers published from 2009 to 2018 in 62 journals spanning the social and behavioural sciences. Authors of 144 (24.0% [95% CI 20.8 - 27.6%]) papers made data available to assess reproducibility, and for 38 others, we obtained source data to reconstruct the dataset. We assessed 143 of the 182 available datasets and found that 76.6 (53.6% [95% CI 45.8 - 60.7%]) papers were rated as precisely reproducible and 105.0 (73.5% [95% CI 66.4 - 80.0%]) papers as at least approximately reproducible (within 15% of the original effects or within .05 of original p-values) after inverse weighting each of the 551 claims by the number of claims per paper. We observed higher reproducibility for papers from Political Science and Economics than other fields, for more recent than older papers, and for papers from journals that required data sharing. Implementing measures to verify that research is reproducible is needed to support trustworthiness in the complex enterprise of knowledge production.

Published claims should be reproducible, yielding the same result when applying the same analysis to the same data. We assessed reproducibility in a stratified random sample of 600 papers published from 2009 to 2018 in 62 journals spanning the social and behavioural sciences. Authors of 144 (24.0% [95% CI 20.8 - 27.6%]) papers made data available to assess reproducibility, and for 38 others, we obtained source data to reconstruct the dataset. We assessed 143 of the 182 available datasets and found that 76.6 (53.6% [95% CI 45.8 - 60.7%]) papers were rated as precisely reproducible and 105.0 (73.5% [95% CI 66.4 - 80.0%]) papers as at least approximately reproducible (within 15% of the original effects or within .05 of original p-values) after inverse weighting each of the 551 claims by the number of claims per paper. We observed higher reproducibility for papers from Political Science and Economics than other fields, for more recent than older papers, and for papers from journals that required data sharing. Implementing measures to verify that research is reproducible is needed to support trustworthiness in the complex enterprise of knowledge production.

Published claims should be reproducible, yielding the same result when applying the same analysis to the same data. We assessed reproducibility in a stratified random sample of 600 papers published from 2009 to 2018 in 62 journals spanning the social and behavioural sciences. Authors of 144 (24.0% [95% CI 20.8 - 27.6%]) papers made data available to assess reproducibility, and for 38 others, we obtained source data to reconstruct the dataset. We assessed 143 of the 182 available datasets and found that 76.6 (53.6% [95% CI 45.8 - 60.7%]) papers were rated as precisely reproducible and 105.0 (73.5% [95% CI 66.4 - 80.0%]) papers as at least approximately reproducible (within 15% of the original effects or within .05 of original p-values) after inverse weighting each of the 551 claims by the number of claims per paper. We observed higher reproducibility for papers from Political Science and Economics than other fields, for more recent than older papers, and for papers from journals that required data sharing. Implementing measures to verify that research is reproducible is needed to support trustworthiness in the complex enterprise of knowledge production.

Published claims should be reproducible, yielding the same result when the same analysis is applied to the same data^1,2. Here we assess reproducibility in a stratified random sample of 600 papers published from 2009 to 2018 in 62 journals spanning the social and behavioural sciences. The authors of 144 (24.0%, 95% confidence interval (CI) = 20.8-27.6%) papers made data available to assess reproducibility and, for 38 others, we obtained source data to reconstruct the dataset. We assessed 143 out of the 182 available datasets and found that 76.6 (53.6%, 95% CI = 45.8-60.7%) papers were rated as precisely reproducible and 105.0 (73.5%, 95% CI = 66.4-80.0%) were rated as at least approximately reproducible (within 15% of the original effects or within 0.05 of original P values) after inverse weighting each of the 551 claims by the number of claims per paper. We observed higher reproducibility for papers from political science and economics compared with other fields, for more recent papers compared with older papers and for papers from journals that require data sharing. Implementation of measures to verify that research is reproducible is needed to support trustworthiness in the complex enterprise of knowledge production^3,4.

Pursuing replicability — independent evidence for previous claims — is important for creating generalizable knowledge^1,2. Here we attempted replications of 274 claims of positive results from 164 quantitative papers published from 2009 to 2018 in 54 journals in the social and behavioural sciences. Replications were high powered on average to detect the original effect size (median of 99.6%), used original materials when relevant and available, and were peer reviewed in advance through a standardized internal protocol. Replications showed statistically significant results in the original pattern for 151 of 274 claims (55.1% (95% confidence interval (CI) 49.2–60.9%)) and for 80.8 of 164 papers (49.3% (95% CI 43.8–54.7%)), weighed for replicating multiple claims per paper. We observed modest variation in replication rates across disciplines (42.5–63.1%), although some estimates had high uncertainty. The median Pearson’s r effect size was 0.25 (95% CI 0.21–0.27) for original studies and 0.10 (95% CI 0.09–0.13) for replication studies, an 82.4% (95% CI 67.8–88.2%) reduction in shared variance. Thirteen methods for evaluating replication success provided estimates ranging from 28.6% to 74.8% (median of 49.3%). Some decline in effect size and significance is expected based on power to detect original effects and regression to the mean because we replicated only positive results. We observe that challenges for replicability extend across social–behavioural sciences, illustrating the importance of identifying conditions that promote or inhibit replicability^3,4.

The opaqueness of author naming and ordering, when coupled with power dynamics, can lead to a number of disadvantages in academic careers. In this commentary, we investigate gender differences in authorship experiences in a large prospective meta-analytic study (k = 46; n = 3,565; 12 countries). We find that women’s and men’s authorship experiences differ significantly with women reporting greater prevalence of problematic behaviors. We present seven actionable recommendations for improving the receipt and reporting of intellectual credit. Such actions are needed to ensure fairness in authorship, which is one of the most powerful factors in academics’ career outcomes.

Replications are important for assessing the reliability of published findings. However, they are costly, and it is infeasible to replicate everything. Accurate, fast, lower-cost alternatives such as eliciting predictions could accelerate assessment for rapid policy implementation in a crisis and help guide a more efficient allocation of scarce replication resources. We elicited judgements from participants on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using an interactive structured elicitation protocol, and we conducted 29 new high-powered replications. After interacting with their peers, participant groups with lower task expertise (‘beginners’) updated their estimates and confidence in their judgements significantly more than groups with greater task expertise (‘experienced’). For experienced individuals, the average accuracy was 0.57 (95% CI: [0.53, 0.61]) after interaction, and they correctly classified 61% of claims; beginners’ average accuracy was 0.58 (95% CI: [0.54, 0.62]), correctly classifying 69% of claims. The difference in accuracy between groups was not statistically significant and their judgements on the full set of claims were correlated (r(98) = 0.48, P < 0.001). These results suggest that both beginners and more-experienced participants using a structured process have some ability to make better-than-chance predictions about the reliability of ‘fast science’ under conditions of high uncertainty. However, given the importance of such assessments for making evidence-based critical decisions in a crisis, more research is required to understand who the right experts in forecasting replicability are and how their judgements ought to be elicited.

People's motivational processes, well-being, and performance are likely to be facilitated through the support of others. Self-determination theory argues that interpersonal supports for autonomy, competence, and relatedness are crucial to achieve these outcomes. In the present study, we provide a comprehensive examination of this formulation based on a meta-analytic database consisting of 4,561 effect sizes from 881 independent samples (N = 443,556). Our results indicate that supports for autonomy, competence, and relatedness were strongly positively related with the satisfaction of these basic needs and strongly negatively related to their frustration. Interpersonal supports for basic needs were strongly positively related with subjective well-being and exhibited small to moderate positive associations with performance. Moderation analyses showed general stability of effects across cultures, although correlations of autonomy support to autonomous motivation weakened as a function of individualism. The opposite pattern was observed for the correlation between relatedness support and intrinsic motivation. Some effects also declined as a function of sample age and lag in measurements. We also find that competence- and relatedness-supportive behaviors explained incremental variance in basic need satisfaction even after controlling for the more established effects of autonomy support. In addition, lateral need supports explained incremental variance in basic need satisfaction after controlling for vertical sources of support. In sum, our results are consistent with the premise that to support optimal motivation, well-being, and performance, a broad set of behaviors that nurture all three basic needs, together with different sources of interpersonal support, should be considered to yield the most benefit. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Replications are important for assessing the reliability of published findings. However, they are costly, and it is infeasible to replicate everything. Accurate, fast, lower-cost alternatives such as eliciting predictions could accelerate assessment for rapid policy implementation in a crisis and help guide a more efficient allocation of scarce replication resources. We elicited judgments from participants on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using an interactive structured elicitation protocol, and we conducted 29 new high-powered replications. After interacting with their peers, participant groups with lower task expertise (‘beginners’) updated their estimates and confidence in their judgements significantly more than groups with greater task expertise (‘experienced’). For experienced individuals, the average accuracy was 0.57 (95% CI: [0.53, 0.61]) after interaction, and they correctly classified 61% of claims; beginners’ average accuracy was 0.58 (95% CI: [0.54, 0.62]), correctly classifying 69% of claims. The difference in accuracy between groups was not statistically significant, and their judgments on the full set of claims were correlated (r(98) = 0.48, p<0.001). These results suggest that both beginners and more experienced participants using a structured process have some ability to make better-than-chance predictions about the reliability of ‘fast science’ under conditions of high uncertainty. However, given the importance of such assessments for making evidence-based critical decisions in a crisis, more research is required to understand who the right experts in forecasting replicability are and how their judgements ought to be elicited.