Joseph Hilgard

Mediation analysis plays a central role in marketing research and consumer psychology by modeling the complex processes underlying total effects. Like other inferential methods, tests of indirect effects are susceptible to false-positive results when multiple defensible analytic decisions can lead to different conclusions (i.e. researcher degrees of freedom). To assess the collective evidential value of mediation analyses in marketing, we (1) measure the relative proximity (RP) of confidence intervals to zero and (2) aggregate a related set of RP scores into a single distribution. Study 1 demonstrates how researcher degrees of freedom in mediation analysis, such as trying many different mediators and outcomes or testing moderated mediation, can inflate false-positive rates to up to 20% in the scenarios we examined. This study also demonstrates the validity of the RP index by examining its correspondence with false-positive findings. In Study 2, we compare the RP distribution of mediation analyses published in top marketing journals (2018-2020) to simulations of null effects, low power, and adequate power. We find that mediation analyses in marketing substantially deviated from simulations with adequate power but were similar to simulated distributions with low power and null effects. We propose study preregistration, increased statistical power, and reduction of selection bias as solutions to increase evidential value of mediation analyses going forward.

Terror management theory postulates that mortality salience (MS) increases the motivation to defend one’s cultural worldviews. How that motivation is expressed may depend on the social norm that is momentarily salient. Meta-analyses were conducted on studies that manipulated MS and social norm salience. Results based on 64 effect sizes for the hypothesized interaction between MS and norm salience revealed a small-to-medium effect of g = 0.34, CI95 [0.26, 0.41]. Bias-adjustment techniques suggested the presence of publication bias and/or the exploitation of researcher degrees of freedom and arrived at smaller effect size estimates for the hypothesized interaction, in several cases reducing the effect to non-significance (range g_corrected = –0.36 to 0.15). To increase confidence in the idea that MS and norm salience interact to influence behavior, preregistered, high-powered experiments using validated norm salience manipulations are necessary. Concomitantly, more specific theorizing is needed to identify reliable boundary conditions of the effect.

In this reply to Bushman et al. (2022), I explain why differences in method and sample demographics are not likely to explain why the obvious effect in Hilgard (2021) is smaller than the indirect effect in Hasan et al (2013). Additionally, Bushman and colleagues claim that the large effects can be explained by accumulation of effects across days. Comparison with longitudinal experimental studies, cross-sectional observational studies, and longitudinal observational studies indicate that this accumulation of effects across three days cannot explain the unparalleled size of effects from Hasan et al. (2013). Accumulation of effects also cannot explain the incredibly large effects observed in single-session studies published by Hasan and colleagues. Better evidence is needed for the claim that other credible studies have reported even larger media violence effects. The data reported in Hasan et al. (2013) are still too good to be true.

We quantitatively reviewed the construct validity evidence for all cognitively based indirect measures of sexual interest in prepubescent children (pedophilic interest) and pubescent children (hebephilic interest) using meta-analysis. Studies were included if they presented scores on a cognitively based indirect measure of pedohebephilic interest for a sample of adolescent or adult males who had committed a sexual offense against a child 16 years of age or younger, or who reported sexual interest in children, and for a comparison group. Studies were also included if they reported on the strength of association between scores on an indirect measure and an independent indicator of pedohebephilic interest in a sample of males. We used meta-analysis with robust variance estimation to summarize effect sizes and metaregression to test potential moderators. Cognitively based indirect measures of pedohebephilic interest showed a moderate difference between pedohebephilic (n = 2,552) and nonpedohebephilic males (n = 2,434), d = 0.61, 95% CI [0.46, 0.76], k = 39. A small-to-moderate correlation was also observed between indirect measures and independent indicators of pedohebephilic interest, r = .23, 95% CI [0.17, 0.28], k = 23, n = 3,623. These effects were qualified by substantial heterogeneity; however, most moderators we tested did not account for a significant amount of heterogeneity. Findings suggest that publication bias did not substantially distort the results. However, the lack of significant moderators suggests more research is needed to understand the conditions under which indirect measures best reflect pedohebephilic interest. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Replication, an important, uncommon, and misunderstood practice, is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understanding to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understanding and observed surprising failures to replicate many published findings. Replication efforts highlighted sociocultural challenges, such as disincentives to conduct replications, framing of replication as personal attack rather than healthy scientific practice, and headwinds for replication contributing to self-correction. Nevertheless, innovation in doing and understanding replication, and its cousins, reproducibility and robustness, have positioned psychology to improve research practices and accelerate progress.

Effect sizes in social psychology are generally not large and are limited by error variance in manipulation and measurement. Effect sizes exceeding these limits are implausible and should be viewed with skepticism. Maximal positive controls, experimental conditions that should show an obvious and predictable effect, can provide estimates of the upper limits of plausible effect sizes on a measure. In this work, maximal positive controls are conducted for three measures of aggressive cognition, and the effect sizes obtained are compared to studies found through systematic review. Questions are raised regarding the plausibility of certain reports with effect sizes comparable to, or in excess of, the effect sizes found in maximal positive controls. Maximal positive controls may provide a means to identify implausible study results at lower cost than direct replication.

Prior research has provided initial evidence that thinking about being excluded by God lowers self-reported well-being in a Dutch sample of Christian students. The current research sought to replicate this finding in two studies. The first experiment recruited a USA sample of Christian students from a secular and religious school. The second experiment recruited a USA online sample of Christians contacted via Mechanical Turk. Results of these two studies replicated the initial finding that thinking about being excluded by God lowers self-reported well-being relative to thinking about being included by God, or contemplating that God created the earth. Moreover, a mini-meta analysis of the original study and the current two studies added the novel insight that thinking about being included by God increased well-being relative to contemplating that God created the earth. Overall, these results show how people’s perceived relationship with God may influence their quality of life.

Interpreting a failure to replicate is complicated by the fact that the failure could be due to the original finding being a false positive, unrecognized moderating influences between the original and replication procedures, or faulty implementation of the procedures in the replication. One strategy to maximize replication quality is involving the original authors in study design. We (N = 17 Labs and N = 1,550 participants, after exclusions) experimentally tested whether original author involvement improved replicability of a classic finding from Terror Management Theory (Greenberg et al., 1994). Our results were non-diagnostic of whether original author involvement improves replicability because we were unable to replicate the finding under any conditions. This suggests that the original finding was either a false positive or the conditions necessary to obtain it are not fully understood or no longer exist. Data, materials, analysis code, preregistration, and supplementary documents can be found on the OSF page: https://osf.io/8ccnw/

Public trust in agricultural biotechnology organizations that produce so-called ‘genetically-modified organisms’ (GMOs) is affected by misinformed attacks on GM technology and worry that producers' concern for profits overrides concern for the public good. In an experiment, we found that reporting that the industry engages in open and transparent research practices increased the perceived trustworthiness of university and corporate organizations involved with GMOs. Universities were considered more trustworthy than corporations overall, supporting prior findings in other technology domains. The results suggest that commitment to, and communication of, open and transparent research practices should be part of the process of implementing agricultural biotechnologies.