Julia M. Rohrer

Collecting a sample that represents the population of interest well constitutes a challenge across the social and behavioural sciences. Psychology in particular frequently relies on convenience samples—most notably students and, increasingly, online participants—with a tendency to either (implicitly) assume representativeness without substantive justification, or to acknowledge a lack of it only in passing. In contrast, researchers rarely engage with the actual implications for their inferences, which undermines the generalisability of psychological findings. Critically, representativeness must be defined with respect to variables relevant to the target of inference, rather than superficial demographic diversity. Here we present the Total Survey Error (TSE) framework as a methodological tool that systematically addresses the multifaceted sources of error—particularly those related to representation—that emerge throughout the research cycle. Although TSE originated in survey research, its principles are broadly applicable to any psychological study seeking inference from sample to population. We offer practical strategies for identifying, preventing, and mitigating representation errors to improve the credibility and generalisability of psychological research.

The city of Leipzig in Germany conducts large-scale school surveys of adolescents in secondary schools. While in 2010 and 2015 girls were somewhat less satisfied than boys, in 2023 this gender gap had doubled. Why? When asking such a reverse causal question, answers often focus on broad narratives, such as the psychological impact of social media. Here, we illustrate how to probe alternative explanations, such as demographic changes and methodological issues. First, we rule out that the observed pattern is a simple scaling artifact. Second, we find that the widening of the gender gap is much more pronounced among students with a migration background. This could plausibly be explained by a shift in the composition of the underlying population, with a strong increase in the proportion of Syrian students, and a relative decrease of Vietnamese students. Third, part of the increasing gender gap could potentially be attributed to survey mode: In 2023, for the first time, the survey was conducted on tablets—and unexpectedly, girls (but not boys) reported significantly lower satisfaction when surveyed on tablet rather than on paper. Lastly, beyond these patterns, we still find significantly widening gender gaps in satisfaction with leisure time activities and relationships to friends. Thus, there may be a substantive increase in the gender gap in satisfaction in those two domains that is not readily attributable to changes in demographics and survey mode.

Psychological researchers usually make sense of regression models by interpreting coefficient estimates directly. This works well enough for simple linear models but is challenging for more complex models with, for example, categorical variables, interactions, nonlinearities, or hierarchical structures. Here, we introduce an alternative approach to making sense of statistical models. The central idea is to abstract away from the mechanics of estimation and to treat models as “counterfactual prediction machines,” which are subsequently queried to estimate quantities and conduct tests that matter substantively. This workflow is model-agnostic; it can be applied in consistent fashion to draw inferences from a wide range of models. We illustrate how to implement this workflow with the marginaleffects package, which supports more than 100 different classes of models in R and Python , and present two worked examples. These examples show how the workflow can be applied across designs (e.g., observational studies, randomized experiments) to answer different research questions (e.g., about associations, causal effects, effect heterogeneity) while facing various challenges (e.g., controlling for confounders in a flexible manner, modeling ordinal outcomes, and interpreting nonlinear models).

Different women experience hormonal contraceptives differently, reporting side effects on their sexuality that range from negative to positive. But research on such causal effects of hormonal contraceptives on psychological outcomes struggles both to identify average causal effects and capture the high heterogeneity in women’s treatment responses. In this study, we leveraged longitudinal data to improve our ability to separate the causal effects of hormonal contraceptives from other sources of association, including observed and unobserved confounding, reverse causality, and attrition. In this programmatic registered report (programmatic registered stage 1 protocol: https://osf.io/kj3h2 ; date of in-principle acceptance: 28/09/2023), we analyzed data from up to 5,041 women (23,130 observations), who participated in PAIRFAM, a German longitudinal panel dataset consisting of 14 waves, using Bayesian multilevel regressions. To deal with confounding and probe the robustness of findings, we implemented two analysis approaches: adjusted regression analysis and inverse probability of treatment weighting approach. We found evidence for positive average treatment effects of hormonal contraceptives on sexual frequency and sexual satisfaction, but no robust evidence for effects on desired sexual frequency. Furthermore, to move beyond average treatment effects, we analyzed heterogeneity in treatment responses. We found relatively high heterogeneity in individual treatment effects on sexual frequency and sexual satisfaction. Interindividual differences were not systematically related to individual treatment effects, and those treatment effects did not predict women’s decisions about which contraceptive method to use in the long run. Our results contribute to understanding the effects of hormonal contraceptives on sexuality in a naturalistic setting, where women adapt their choice of contraceptive method to their own experiences.

Interest in emotional variability as an interindividual difference is growing. Yet, basic features of the construct, such as whether it can be measured reliably and whether interindividual differences remain stable over time, have not yet been investigated extensively. To improve our understanding of the construct, we examined two longitudinal data sets, each comprising two waves of daily assessments, one with a 3-month and the other with a 16-month retest interval. Overcoming a key methodological limitation of past approaches, we used Bayesian censored location scale models as an alternative modeling approach that accounts for biases introduced by bounded rating scales. The results showed that the variability estimates from the models had reliabilities around rel = .64, which can be sufficient for group-level predictions. Additionally, the latent stability of r = .60 provides evidence for stable individual differences in emotional variability, suggesting that it is more than just a transient state. Our findings partially confirm previous studies, but also suggest that the reliability and stability of intraindividual emotional variability may have been underestimated because of biases due to censored distributions, something that has often been neglected in past work. We ran exploratory analyses to further examine the influence of external events and individual life transitions on emotional variability and found that emotional variability was potentially responsive to environmental changes due to the transition from studying to working life in one data set, and due to the COVID-19 pandemic in the second data set.

Psychological researchers are interested in how things change over time and routinely make claims about age effects (e.g., personality maturation), cohort effects (e.g., generational differences in narcissism), and sometimes, period effects (e.g., secular trends in mental health). The age-period-cohort identification problem means that these claims are not possible based on the data alone: Any possible temporal pattern can be explained by an infinite number of combinations of age, period, and cohort effects. This concern holds regardless of the study design (it also applies to longitudinal designs covering multiple cohorts) and the number of observations available (it also applies if researchers observe the whole population). Researchers usually rely on statistical models that impose constraints to pick one specific decomposition of effects. Unfortunately, these constraints often remain opaque, resulting in a lack of scrutiny of the underlying assumptions. How can researchers reason more transparently and systematically about age, period, and cohort? Here, I summarize advances in the understanding of the precise nature of the identification problem, provide an overview of ways to move forward, and highlight one approach that is particularly transparent about assumptions: bounding analysis, a framework developed by sociologists Ethan Fosse and Christopher Winship. To illustrate this approach, I analyze how age, period, and cohort affect attitudes toward working mothers in the German General Social Survey.