Ruben C. Arslan

Serves as a companion toolkit for the 'formr' survey framework (). The package acts as a bridge between a 'formr' server and a local R environment. Key features include an API client for fetching, type-casting, and automatically scoring data; a project management workflow for syncing study assets (surveys, CSS) for local editing; and functions for use within 'formr' runs to generate dynamic, personalized feedback plots and to simplify survey logic.

Valid generalisations from sample to population require representative samples. This is especially the case for psychological test norms as they are only useful to the extent that the normative samples can stand in for reference populations. However, as a field, we continue to face difficulties defining, acquiring, and describing representative samples. Having engaged with similar problems for a long time, political and other social scientists have developed a sophisticated set of tools for dealing with them. In this article, we therefore draw on insights from these fields to illustrate what measures need to be taken at all research stages to maximise sample representativeness, and the potential benefits of sample weighting in psychology. We argue that a specific class of weighting methods, Regularised Prediction and Poststratification (RPP) holds promise for improving psychological test norming practice. RPP involves fitting a regularised prediction model to a large and diverse sample and weighting predicted scores by true population values (sourced from census data, for example). Using IQ test data from the TwinLife study (N = 9,980, Culture Fair Test, CFT 20-R) as a case study, we show that RPP yields IQ scores that differ by up to 19 IQ points (on average by 4.19 points) from those reported in the CFT 20-R manual, which were based on traditional norming methods. We additionally observed sizeable differences to other norming/weighting approaches, such as semiparametric continuous norming and using weights provided by the sample provider . Differences of this magnitude can lead to decision errors with potentially life-altering consequences (e.g., about intellectual disability). Beyond individual diagnostics, we contend that psychology could benefit from more widely available, high-quality norms for many aspects of research and practice and plead for the change in research practice necessary to make these benefits possible.

Different women experience hormonal contraceptives differently, reporting side effects on their sexuality that range from negative to positive. But research on such causal effects of hormonal contraceptives on psychological outcomes struggles both to identify average causal effects and capture the high heterogeneity in women’s treatment responses. In this study, we leveraged longitudinal data to improve our ability to separate the causal effects of hormonal contraceptives from other sources of association, including observed and unobserved confounding, reverse causality, and attrition. In this programmatic registered report (programmatic registered stage 1 protocol: https://osf.io/kj3h2 ; date of in-principle acceptance: 28/09/2023), we analyzed data from up to 5,041 women (23,130 observations), who participated in PAIRFAM, a German longitudinal panel dataset consisting of 14 waves, using Bayesian multilevel regressions. To deal with confounding and probe the robustness of findings, we implemented two analysis approaches: adjusted regression analysis and inverse probability of treatment weighting approach. We found evidence for positive average treatment effects of hormonal contraceptives on sexual frequency and sexual satisfaction, but no robust evidence for effects on desired sexual frequency. Furthermore, to move beyond average treatment effects, we analyzed heterogeneity in treatment responses. We found relatively high heterogeneity in individual treatment effects on sexual frequency and sexual satisfaction. Interindividual differences were not systematically related to individual treatment effects, and those treatment effects did not predict women’s decisions about which contraceptive method to use in the long run. Our results contribute to understanding the effects of hormonal contraceptives on sexuality in a naturalistic setting, where women adapt their choice of contraceptive method to their own experiences.

Interest in emotional variability as an interindividual difference is growing. Yet, basic features of the construct, such as whether it can be measured reliably and whether interindividual differences remain stable over time, have not yet been investigated extensively. To improve our understanding of the construct, we examined two longitudinal data sets, each comprising two waves of daily assessments, one with a 3-month and the other with a 16-month retest interval. Overcoming a key methodological limitation of past approaches, we used Bayesian censored location scale models as an alternative modeling approach that accounts for biases introduced by bounded rating scales. The results showed that the variability estimates from the models had reliabilities around rel = .64, which can be sufficient for group-level predictions. Additionally, the latent stability of r = .60 provides evidence for stable individual differences in emotional variability, suggesting that it is more than just a transient state. Our findings partially confirm previous studies, but also suggest that the reliability and stability of intraindividual emotional variability may have been underestimated because of biases due to censored distributions, something that has often been neglected in past work. We ran exploratory analyses to further examine the influence of external events and individual life transitions on emotional variability and found that emotional variability was potentially responsive to environmental changes due to the transition from studying to working life in one data set, and due to the COVID-19 pandemic in the second data set.

Many behavioral scientists do not agree on core constructs and how they should be measured. Different literatures measure related constructs, but the connections are not always obvious to readers and meta-analysts. Many measures in behavioral science are based on agreement with survey items. Because these items are sentences, computerized language models can make connections between disparate measures and constructs and help researchers regain an overview over the rapidly growing, fragmented literature. Our fine-tuned language model, the SurveyBot3000, accurately predicts the correlations between survey items, the reliability of aggregated measurement scales, and intercorrelations between scales from item positions in semantic vector space. We measured the model’s performance as the convergence between its synthetic model estimates and empirical coefficients observed in human data. In our pilot study, the out-of-sample accuracy was .71 for item correlations, .89 for reliabilities, and .89 for scale correlations. In our preregistered validation study using novel items, the out-of-sample accuracy was slightly reduced to .59 for item correlations, .84 for reliabilities, and .84 for scale correlations. The synthetic item correlations showed an average prediction error of .17, and there were larger errors for middling correlations. Predictions exhibited generalizability beyond the training data and across various domains, with some variability in accuracy. Our work shows language models can reliably predict psychometric relationships between survey items, enabling researchers to evaluate new measures against existing scales, reduce redundancy in measurement, and work toward a more unified behavioral-science taxonomy.

We examined the extent to which constructs and measures have proliferated in psychological science. We integrated two large databases obtained from the American Psychology Association that have been used to keep track of constructs, measures, and research in the psychological-science literature for the past 30 years. In our descriptive analyses, we found that (a) thousands of new constructs and measures are published each year, (b) most measures are used very few times, and (c) there is no trend toward consensus or standardization in the use of constructs and measures; in fact, there is a trend toward even greater fragmentation over time. That is, constructs and measures are proliferating. We conclude that measurement in the psychological-science literature is fragmented, creating problems such as redundancy and confusion and stifling cumulative scientific progress. We conclude by providing suggestions for what researchers can do about this problem.

Cronbach’s α is the most widely reported metric of the reliability of psychological measures. Decisions about an observed α’s adequacy are often made using rule-of-thumb thresholds, such as α of at least .70. Such thresholds can put pressure on researchers to make their measures meet these criteria, similar to the pressure to meet the significance threshold with p values. We examined whether α values reported in the psychology literature are inflated at the rule-of-thumb thresholds (αs = .70, .80, .90) because of, for example, overfitting to in-sample data (α-hacking) or publication bias. We extracted reported α values from three very large data sets covering the general psychology literature (> 30,000 α values taken from > 74,000 published articles in American Psychological Association [APA] journals), the industrial and organizational (I/O) psychology literature (> 89,000 α values taken from > 14,000 published articles in I/O journals), and the APA’s PsycTests database, which aims to cover all psychological measures published since 1894 (> 67,000 α values taken from > 60,000 measures). The distributions of these values show robust evidence of excesses at the α = .70 rule-of-thumb threshold that cannot be explained by justifiable measurement practices. We discuss the scope, causes, and consequences of α-hacking and how increased transparency, preregistration of measurement strategy, and standardized protocols could mitigate this problem.