Jesse J. Chandler

We attempted replications of 274 claims of positive results from 164 quantitative papers published from 2009 to 2018 in 54 journals in the social and behavioral sciences. Replications were high-powered on average to detect the original effect size (Median = 99.6%), used original materials when relevant and available, and were peer-reviewed in advance through a standardized internal protocol. Replications showed statistically significant results in the original pattern for 151 of 274 claims (55.1% [95% CI 49.2 - 60.9%]) and for 80.8 of 164 papers (49.3% [95% CI 43.8 - 54.7%]) weighed for replicating multiple claims per paper. We observed modest variation in replication rates across disciplines (42.5% to 63.1%) though some estimates had high uncertainty. For claims with Pearson’s r effect sizes, the median was 0.25 [95% CI 0.21 - 0.27] for original studies and 0.10 [95% CI 0.09 - 0.13] for replication studies, a 82.4% [95% CI 67.8 - 88.2%] reduction in shared variance. Thirteen methods for evaluating replication success provided estimates ranging from 28.6% to 74.8% (median = 49.3%). Some decline in effect size and significance is expected based on power to detect original effects and regression to the mean due to replicating only positive results. The conditions that promote or inhibit replicability are worthy of additional investigation.

The same dataset can be analysed in different justifiable ways to answer the same research question, potentially challenging the robustness of empirical science^1-3. In this crowd initiative, we investigated the degree to which research findings in the social and behavioural sciences are contingent on analysts' choices. We examined a stratified random sample of 100 studies published between 2009 and 2018, in which, for one claim per study, at least five reanalysts independently reanalysed the original data. The statistical appropriateness of the reanalyses was assessed in peer evaluations, and the robustness indicators were inspected along a range of research characteristics and study designs. We found that 34% of the independent reanalyses yielded the same result (within a tolerance region of ±0.05 Cohen's d) as the original report; with a four times broader tolerance region, this indicator increased to 57%. Of the reanalyses conducted, 74% reached the same conclusion as the original investigation, 24% yielded no effects or inconclusive results and 2% reported the opposite effect. This exploratory study indicates that the common single-path analyses in social and behavioural research should not be simply assumed to be robust to alternative analyses⁴. Therefore, we recommend the development and use of practices to explore and communicate this neglected source of uncertainty.

Pursuing replicability — independent evidence for previous claims — is important for creating generalizable knowledge^1,2. Here we attempted replications of 274 claims of positive results from 164 quantitative papers published from 2009 to 2018 in 54 journals in the social and behavioural sciences. Replications were high powered on average to detect the original effect size (median of 99.6%), used original materials when relevant and available, and were peer reviewed in advance through a standardized internal protocol. Replications showed statistically significant results in the original pattern for 151 of 274 claims (55.1% (95% confidence interval (CI) 49.2–60.9%)) and for 80.8 of 164 papers (49.3% (95% CI 43.8–54.7%)), weighed for replicating multiple claims per paper. We observed modest variation in replication rates across disciplines (42.5–63.1%), although some estimates had high uncertainty. The median Pearson’s r effect size was 0.25 (95% CI 0.21–0.27) for original studies and 0.10 (95% CI 0.09–0.13) for replication studies, an 82.4% (95% CI 67.8–88.2%) reduction in shared variance. Thirteen methods for evaluating replication success provided estimates ranging from 28.6% to 74.8% (median of 49.3%). Some decline in effect size and significance is expected based on power to detect original effects and regression to the mean because we replicated only positive results. We observe that challenges for replicability extend across social–behavioural sciences, illustrating the importance of identifying conditions that promote or inhibit replicability^3,4.

The same dataset can be analysed in different justifiable ways to answer the same research question, potentially challenging the robustness of empirical science1–3. In this crowd initiative, we investigated the degree to which research findings in the social and behavioural sciences are contingent on analysts’ choices. We examined a stratified random sample of 100 studies published between 2009 and 2018, where for one claim per study, at least five re-analysts independently re-analysed the original data. The statistical appropriateness of the re-analyses was assessed in peer evaluations, and the robustness indicators were inspected along a range of research characteristics and study designs. We found that 34% of the independent re-analyses yielded the same result (within a tolerance region of +/- 0.05 Cohen’s d) as the original report; with a four times broader tolerance region, this indicator rose to 57%. Regarding the conclusions drawn, 74% of analyses were reported to arrive at the same conclusion as in the original investigation; 24% to no effects/inconclusive result, and 2% to the opposite effect as in the original investigation. This exploratory study suggests that the common single-path analyses in social and behavioural research should not simply be assumed to be robust to alternative analyses4. Therefore, we recommend the development and use of practices to explore and communicate this neglected source of uncertainty.

A substantial fraction of the observational and experimental data collected in the social sciences is now coming from crowdsourcing platforms, and being acquainted with the characteristics and dynamics of such samples is more important than ever. This chapter aims to familiarize computational social scientists with crowdsourced samples, in particular with what has become the most popular sampling frame in the social sciences: Amazon Mechanical Turk (MTurk). The chapter answers several questions: Who joins the MTurk platform as a worker and why? How do MTurk samples compare to traditional samples in survey and experimental research, such as undergraduate students? What challenges (e.g., for data quality) can researchers expect to face on MTurk? What are the ethical concerns that these online workplaces bring to the surface? We hope the answers to these questions will be useful to a variety of computational social scientists: aspiring survey and experimental researchers who want to familiarize themselves with the main characteristics of crowdsourcing; experienced researchers who seek a deeper understanding of the implications of their methodological choices; and outsiders who want to get a peek at how social science is conducted now and will be conducted in the future.

This dataset is from the Many Labs Replication Project [1] in which 13 effects were replicated across 36 samples and over 6,000 participants. Data from the replications are included, along with demographic variables about the participants and contextual information about the environment in which the replication was conducted. Data were collected in-lab and online through a standardized procedure administered via an online link. The dataset is stored on the Open Science Framework website. These data could be used to further investigate the results of the included 13 effects or to study replication and generalizability more broadly.

This manuscript contains our responses to several commentaries about the Many Labs Project (Klein et al., 2014).

Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of thirteen classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, ten effects replicated consistently. One effect – imagined contact reducing prejudice – showed weak support for replicability. And two effects – flag priming influencing conservatism and currency priming influencing system justification – did not replicate. We compared whether the conditions such as lab versus online or U.S. versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.

A strong participant recruitment plan is a major determinant of the success of human subjects research. The plan adopted by researchers will determine the kinds of inferences that follow from the collected data and how much it will cost to collect. Research studies with weak or nonexistent recruitment plans risk recruiting too few participants or the wrong kind of participants to be able to answer the question that motivated them. This chapter outlines key considerations for researchers who are developing recruitment plans and provides suggestions for how to make recruiting more efficient.

Despite their prevalence in the marketplace, little empirical attention has been paid to how employee uniforms affect consumer reactions to service experiences. We propose that employee uniforms facilitate the shared categorization of employees and their organization in the mind of the customer, which affects many of the inferences that customers draw following service encounters. Study 1 shows that uniforms lead to greater attribution of responsibility to the company for employee behavior, especially following poor service. Studies 2 and 3 show that uniforms also lead to more assimilation of judgments across employees, increasing the impact of one employee's behavior on judgments of other employees of the same organization. Study 3 shows that employee uniforms lead to more extreme judgments of the company following service encounters. It also shows that bad (good) service from a uniformed employee makes competing companies look better (worse), indicating that uniforms can elicit contrast effects across companies. In sum, the mere presence of a uniform on an unsatisfactory service or retail employee can damage judgments of the organization and its employees and improve judgments of rival organizations compared to identical service from a nonuniformed employee. Managers seem unaware of these negative consequences. These same principles are likely to apply to a wide variety of uniformed services, including police, military, firefighters, and health‐care providers.