Rink Hoekstra

To correct errors in the scientific record, researchers must dedicate time and attention to the task. However, time and attention are both extremely limited resources in science. Resource-efficient quality control that targets the scientific output most in need of correction is a crucial part of facilitating effective scientific self-correction in practice. This piece summarizes a one-day hackathon that took place during the Perspectives on Scientific Error conference in 2024. The hackathon was devoted to discussing how scientific fields such as psychology can coordinate scientific quality control work in practice in three situations: 1) for promotions and getting tenure, 2) at journals during pre- and post-publication stages, and 3) in error detection projects. These hackathon proceedings contain a summary of these three discussions, as well as a summary of topics brought up during the general discussion (see end of piece). We believe this can serve as a starting point for further discussion about how to select studies to scrutinize during pre-, and post-publication stages within and across different fields.Update November 2026: we have uploaded a version 2 of this preprint, now including an author note stating that: This preprint was submitted to the Journal of Trial and Error, an independent diamond open-access journal, and received peer review. Although we have decided not to pursue journal publication, we are grateful to the reviewers and editor for their thoughtful engagement with the work. We value the transparency that preprinting makes possible and, with the permission of both reviewers and the editor (Sarahanne Field), have included the named reviews as an appendix. We hope that sharing the reviews alongside the preprint preserves and contributes to the broader scholarly conversation this work was intended to stimulate, rather than incorporating feedback from an otherwise private review process into a revised manuscript.

To correct errors in the scientific record, researchers must dedicate time and attention to the task. However, time and attention are both extremely limited resources in science. Resource-efficient quality control that targets the scientific output most in need of correction is a crucial part of facilitating effective scientific self-correction in practice. This piece summarizes a one-day hackathon that took place during the Perspectives on Scientific Error conference in 2024. The hackathon was devoted to discussing how scientific fields such as psychology can coordinate scientific quality control work in practice in three situations: 1) for promotions and getting tenure, 2) at journals during pre- and post-publication stages, and 3) in error detection projects. These hackathon proceedings contain a summary of these three discussions, as well as a summary of topics brought up during the general discussion (see end of piece). We believe this can serve as a starting point for further discussion about how to select studies to scrutinize during pre-, and post-publication stages within and across different fields.Update November 2026: we have uploaded a version 2 of this preprint, now including an author note stating that: This preprint was submitted to the Journal of Trial and Error, an independent diamond open-access journal, and received peer review. Although we have decided not to pursue journal publication, we are grateful to the reviewers and editor for their thoughtful engagement with the work. We value the transparency that preprinting makes possible and, with the permission of both reviewers and the editor (Sarahanne Field), have included the named reviews as an appendix. We hope that sharing the reviews alongside the preprint preserves and contributes to the broader scholarly conversation this work was intended to stimulate, rather than incorporating feedback from an otherwise private review process into a revised manuscript.

Numerous solutions have been proposed to address the problem of statistical misuse and misunderstanding in research in psychology. This might have diverted attention from how or whether scientists are actually engaging in statistical reasoning by arguing about and with statistics within the context of the reasoning involved in empirical research in their written papers. In this paper, we studied statistical reasonings in psychological research papers. Specifically, we inductively coded arguments of 42 papers from seven highly cited psychology journals. With these data, we formed a partial, empirical typology with which we described different types of statistical arguments used in the literature. Papers, and their statistical arguments, in empirical psychology can be interpreted as possessing a ‘legal’ nature. Common arguments, such as statistical significance, are often used as a tacit and concise way of certifying that a claim, including conclusions, is empirically and statistically substantiated under the social standards of good research in our field. However, approval of research based on certifications may hinder the exposition and clarity of statistical arguments.

The gap between research and clinical practice, known as the scientist-practitioner gap, poses a persistent challenge in clinical psychology. Although existing research suggests that effective engagement with research can be challenging for clinicians, there is limited understanding of the specific barriers, obstacles, and potential solutions as perceived by clinicians themselves. In this focus group study, we explored junior clinicians' perceptions, attitudes, and interactions with clinical research, specifically targeting Dutch mental health psychologists following postgraduate training (n = 32). Through reflexive thematic analysis, we investigated their relation to research in clinical psychology, their perceived value of research, barriers to engagement, and suggestions for the future. Clinicians reported multiple barriers including difficulties in accessing, understanding, and applying research in practice, as well as skepticism about its relevance. We summarize clinicians’ personal opinions and motivations regarding research, systemic issues such as insufficient organizational support, research priorities not matching demands in practice, and methodological limitations of conventional psychotherapy research. With this study, we stressed the need for enhanced communication between researchers and clinicians, methodological considerations, and greater inclusion of clinicians’ perspectives in research processes.

The repliCATS-SCORE dataset consists of five human elicitation studies predicting the replicability of published research in the social and behavioural sciences, conducted by the repliCATS project (Collaborative Assessments for Trustworthy Science) as part of DARPA’s Systematizing Confidence in Open Research and Evidence or SCORE program (2019-2022). The human elicitations were conducted using a structured group deliberation technique – the IDEA protocol – for a total of 4000 published research claims and an additional 25 claim pilot study. The dataset for each study includes participant-level judgements across two elicitation rounds, mathematically aggregated group judgements or “confidence scores” calculated using a suite of 38 aggregation methods, as well as participant free-text reasoning. The repliCATS-SCORE dataset can be re-used in support of studies interrogating scholarly judgements of replicability, plausibility and overall credibility, post-publication peer review and critical evaluation training, developing novel mathematical aggregation models that forecast replication outcomes, and comparative studies.

In recent years, the advancement of open science has led to data sharing becoming more common practice. Data availability has clear merits for science because it opens up possibilities for reuse of data sets by others, leading to less redundancy, more efficiency, and more transparency. The ideal is for scientific data to be as open as possible and findable, accessible, interoperable, and reusable (FAIR). Parallel to this development, recent times have seen more stringent guidelines with respect to data privacy, culminating in the General Data Protection Regulation law. Navigating the balance between protecting participants’ privacy and making one’s data set as open as possible can be challenging for researchers. In this article, we provide two worked examples with real data sets from the behavioral and social sciences on how to be as open as possible and as closed as necessary with the goal of maximally facilitating science while minimizing the risk of participant identification.

In recent years, the advancement of open science has led to data sharing becoming more common practice. Data availability has clear merits for science as it opens up possibilities for re-use of datasets by others, leading to less redundancy, more efficiency, and more transparency. The ideal is for scientific data to be as open as possible and FAIR: Findable, Accessible, Interoperable, and Reusable. Parallel to this development, recent times have seen more stringent guidelines with respect to data privacy, culminating in the General Data Protection Regulation law, or GDPR. Navigating the balance between protecting participants’ privacy and making one's dataset as open as possible can be challenging for researchers. In this paper, we provide two worked examples with real datasets from the behavioral and social sciences on how to be as open as possible and as closed as necessary, with the goal of maximally facilitating science while minimizing the risk of participant identification.

In response to the replication and confidence crisis across various empirical disciplines, ensuring the validity of research has gained attention. High validity is crucial for obtaining replicable and robust study outcomes when both exploring new questions and replicating previous findings. In this study, we aimed to address this issue by developing a comprehensive checklist to assist researchers in enhancing and monitoring the validity of their research. After systematically analyzing previous findings on validity, a comprehensive list of potential checklist items was compiled. Over the course of three rounds, more than 30 interdisciplinary and psychological-science experts participated in a Delphi study. Experts rated items on their importance and were given the opportunity to propose novel items as well as improve existing ones. This process resulted in a final set of 91 items, organized according to common stages of a research project. The VALID checklist is accessible online ( https://www.validchecklist.com/ ) and provides researchers with an adaptable, versatile tool to monitor and improve the validity of their research and to suit their specific needs. By focusing on adaptiveness during its development, VALID encompasses 331 unique checklist versions, making it a one-stop solution suitable for a wide range of projects, designs, and requirements.

Most researchers receive formal training in frequentist statistics during their undergraduate studies. In particular, hypothesis testing is usually rooted on the null hypothesis significance testing paradigm and its p‐value. Null hypothesis Bayesian testing and its so‐called Bayes factor are now becoming increasingly popular. Although the Bayes factor is often introduced as being the Bayesian counterpart to the p‐value, its computation, use, and interpretation are quite distinct from the p‐value. There is now evidence confirming that the application of the Bayes factor in applied research is ill‐devised. To improve the current status quo, we have created a Shiny/R app called the Bayes factor, which offers a dynamic tutorial for learning all the basics about the Bayes factor. In this paper, we explain how the app works and we offer suggestions on how to use it in class or self‐study settings. The app is freely available at https://statsedge.org/shiny/LearnBF/.