Thomas Pfeiffer
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Thomas Pfeiffer has a total of 886 co-authors. Here are the top co-authors by number of shared publications:
Anna Dreber
18 shared publications- Magnus Johannesson
Department of Economics, Stockholm School of Economics, Stockholm, Sweden
15 shared publications Yiling Chen
15 shared publications- Felix Holzmeister
Department of Economics, University of Innsbruck, Innsbruck, Austria
13 shared publications Magnus Johannesson
13 shared publications- Anna Dreber
Department of Economics, Stockholm School of Economics, Stockholm, Sweden
10 shared publications
Latest Publications
- Open Access
- 15 October 2026
- Published by Center for Open Science in MetaArXiv
Large-scale human predictions of the replicability of published social and behavioural science papers – a multi- study analysis
This paper reports two approaches to forecasting replicability for a corpus of 3000 published social science papers, using large-scale human assessments. Replication Markets used incentivized surveys and prediction markets where participants traded assets linked to replication outcomes and market prices were interpreted as replication predictions. The repliCATS project used a structured group deliberation protocol, including interactive discussion, and mathematically aggregated forecasts to generate replication predictions. Accuracy for both approaches was validated against a subset of (n=37) independent, high-power replication studies. The predictive accuracy (median [range]) achieved for AUC by Replication Markets was 0.76 [0.72-0.82], and 0.76 [0.68-0.81] by repliCATS. Replication Markets achieved a classification accuracy of 73% [68-76%] and repliCATS achieved 68% [59-78%]. These results place the performance of both teams within the accuracy range achieved in prior replication forecasting studies. We conclude that informative forecasts can be elicited by both methods, but there are trade-offs between scale and accuracy.
- 15 October 2026
- Published by Center for Open Science in MetaArXiv
Open AccessLarge-scale human predictions of the replicability of published social and behavioural science papers – a multi- study analysis
This paper reports two approaches to forecasting replicability for a corpus of 3000 published social science papers, using large-scale human assessments. Replication Markets used incentivized surveys and prediction markets where participants traded assets linked to replication outcomes and market prices were interpreted as replication predictions. The repliCATS project used a structured group deliberation protocol, including interactive discussion, and mathematically aggregated forecasts to generate replication predictions. Accuracy for both approaches was validated against a subset of (n=37) independent, high-power replication studies. The predictive accuracy (median [range]) achieved for AUC by Replication Markets was 0.76 [0.72-0.82], and 0.76 [0.68-0.81] by repliCATS. Replication Markets achieved a classification accuracy of 73% [68-76%] and repliCATS achieved 68% [59-78%]. These results place the performance of both teams within the accuracy range achieved in prior replication forecasting studies. We conclude that informative forecasts can be elicited by both methods, but there are trade-offs between scale and accuracy.
- Open Access
- 1 April 2026
- Published by Center for Open Science in MetaArXiv
Assessments of Credibility in the Social and Behavioral Sciences
Credibility assessment — determining whether research findings are trustworthy or believable — is essential to the research process. One aspect of credibility is repeatability, which includes assessing whether consistent results are obtained when using new data to answer the same question (replicability), when repeating the original analyses with the original data (reproducibility), or when conducting alternative analyses about the same question with the original data (robustness). These features of repeatability differ in the resources required to investigate them, and it is unknown how they relate with one another and with other features of credibility. We investigated relationships among credibility measures in a stratified random sample of claims made across the social and behavioral sciences. Measures of repeatability were modestly correlated with each other (r’s = 0.30, -0.04, -0.23) though the correlation between robustness and reproducibility is likely misestimated because selecting claims for robustness testing was partly contingent on reproducibility success. Replicability and human and machine predictions of replicability were modestly correlated (Median r = 0.23; Range = -0.10 to 0.47). Though estimated with substantial uncertainty in some cases, no discipline showed consistently higher repeatability than other disciplines across measures. For example, Education had the highest replicability estimate (0.63, 95% CI [.32 - .86]) and the lowest reproducibility estimate (0.25, 95% CI [.25, 95% CI [.06 - .38]), whereas Economics had the lowest replicability estimate (0.43, 95% CI [.28 - .64]) and nearly the highest reproducibility estimate (0.70, 95% CI [.56 - .85]). Repeatability measures were modestly and heterogeneously associated with other potential indicators of credibility (Median |r| = 0.08; Range = 0.01 to 0.55). Credibility assessment is multidimensional with substantial opportunity for innovation and validation of its measurement.
- 1 April 2026
- Published by Center for Open Science in MetaArXiv
Open AccessAssessments of Credibility in the Social and Behavioral Sciences
Credibility assessment — determining whether research findings are trustworthy or believable — is essential to the research process. One aspect of credibility is repeatability, which includes assessing whether consistent results are obtained when using new data to answer the same question (replicability), when repeating the original analyses with the original data (reproducibility), or when conducting alternative analyses about the same question with the original data (robustness). These features of repeatability differ in the resources required to investigate them, and it is unknown how they relate with one another and with other features of credibility. We investigated relationships among credibility measures in a stratified random sample of claims made across the social and behavioral sciences. Measures of repeatability were modestly correlated with each other (r’s = 0.30, -0.04, -0.23) though the correlation between robustness and reproducibility is likely misestimated because selecting claims for robustness testing was partly contingent on reproducibility success. Replicability and human and machine predictions of replicability were modestly correlated (Median r = 0.23; Range = -0.10 to 0.47). Though estimated with substantial uncertainty in some cases, no discipline showed consistently higher repeatability than other disciplines across measures. For example, Education had the highest replicability estimate (0.63, 95% CI [.32 - .86]) and the lowest reproducibility estimate (0.25, 95% CI [.25, 95% CI [.06 - .38]), whereas Economics had the lowest replicability estimate (0.43, 95% CI [.28 - .64]) and nearly the highest reproducibility estimate (0.70, 95% CI [.56 - .85]). Repeatability measures were modestly and heterogeneously associated with other potential indicators of credibility (Median |r| = 0.08; Range = 0.01 to 0.55). Credibility assessment is multidimensional with substantial opportunity for innovation and validation of its measurement.
- Open Access
- 1 April 2026
- Published by Center for Open Science in MetaArXiv
Large-scale human predictions of the replicability of published social and behavioural science papers – a multi- study analysis
This paper reports two approaches to forecasting replicability for a corpus of 3000 published social science papers, using large-scale human assessments. Replication Markets used incentivized surveys and prediction markets where participants traded assets linked to replication outcomes and market prices were interpreted as replication predictions. The repliCATS project used a structured group deliberation protocol, including interactive discussion, and mathematically aggregated forecasts to generate replication predictions. Accuracy for both approaches was validated against a subset of (n=37) independent, high-power replication studies. The predictive accuracy (median [range]) achieved for AUC by Replication Markets was 0.76 [0.72-0.82], and 0.76 [0.68-0.81] by repliCATS. Replication Markets achieved a classification accuracy of 73% [68-76%] and repliCATS achieved 68% [59-78%]. These results place the performance of both teams within the accuracy range achieved in prior replication forecasting studies. We conclude that informative forecasts can be elicited by both methods, but there are trade-offs between scale and accuracy.
- 1 April 2026
- Published by Center for Open Science in MetaArXiv
Open AccessLarge-scale human predictions of the replicability of published social and behavioural science papers – a multi- study analysis
This paper reports two approaches to forecasting replicability for a corpus of 3000 published social science papers, using large-scale human assessments. Replication Markets used incentivized surveys and prediction markets where participants traded assets linked to replication outcomes and market prices were interpreted as replication predictions. The repliCATS project used a structured group deliberation protocol, including interactive discussion, and mathematically aggregated forecasts to generate replication predictions. Accuracy for both approaches was validated against a subset of (n=37) independent, high-power replication studies. The predictive accuracy (median [range]) achieved for AUC by Replication Markets was 0.76 [0.72-0.82], and 0.76 [0.68-0.81] by repliCATS. Replication Markets achieved a classification accuracy of 73% [68-76%] and repliCATS achieved 68% [59-78%]. These results place the performance of both teams within the accuracy range achieved in prior replication forecasting studies. We conclude that informative forecasts can be elicited by both methods, but there are trade-offs between scale and accuracy.
- Open Access
- 21 January 2026
- Published by The Royal Society in Royal Society Open Science
- Vol. 13 (1)
Using prediction markets and forecasting surveys to predict 28 replication outcomes of classic articles in social psychology and judgement and decision making
Can researchers predict if classic findings published in the field of social psychology and judgement and decision making replicate? We set up prediction markets and a forecasting survey for predicting replications of 28 experiments of classic well-cited articles. Forecasters predicted if the original results would replicate, where a successful replication was defined as an effect in the same direction as the original and a signal (p-value lower than 0.05). Of the 28 original studies, 16 (57%) met the replication success criteria, compared to a predicted replication rate of 70% in the prediction markets and 65% replication rate in the survey. We concluded only suggestive evidence for associations of replication outcomes with prediction market prices (r = 0.43, 95% CI [0.07, 0.69]) and average survey beliefs (r = 0.26, 95% CI [–0.12, 0.58]). The prediction market effects were similar to observed effects in previous prediction market studies and suggest that prediction markets can to some extent predict replication outcomes, yet predictions are far from perfect and conducting replications is much more informative about the credibility of published findings. Data and code are available at https://doi.org/10.17605/OSF.IO/2KMH7.
- 21 January 2026
- Published by The Royal Society in Royal Society Open Science
- Vol. 13 (1)
Open AccessUsing prediction markets and forecasting surveys to predict 28 replication outcomes of classic articles in social psychology and judgement and decision making
Can researchers predict if classic findings published in the field of social psychology and judgement and decision making replicate? We set up prediction markets and a forecasting survey for predicting replications of 28 experiments of classic well-cited articles. Forecasters predicted if the original results would replicate, where a successful replication was defined as an effect in the same direction as the original and a signal (p-value lower than 0.05). Of the 28 original studies, 16 (57%) met the replication success criteria, compared to a predicted replication rate of 70% in the prediction markets and 65% replication rate in the survey. We concluded only suggestive evidence for associations of replication outcomes with prediction market prices (r = 0.43, 95% CI [0.07, 0.69]) and average survey beliefs (r = 0.26, 95% CI [–0.12, 0.58]). The prediction market effects were similar to observed effects in previous prediction market studies and suggest that prediction markets can to some extent predict replication outcomes, yet predictions are far from perfect and conducting replications is much more informative about the credibility of published findings. Data and code are available at https://doi.org/10.17605/OSF.IO/2KMH7.
- Open Access
- 20 December 2024
- Published by Springer Nature in Nature Human Behaviour
- Vol. 9 (2) , 287-304
Predicting the replicability of social and behavioural science claims in COVID-19 preprints
Replications are important for assessing the reliability of published findings. However, they are costly, and it is infeasible to replicate everything. Accurate, fast, lower-cost alternatives such as eliciting predictions could accelerate assessment for rapid policy implementation in a crisis and help guide a more efficient allocation of scarce replication resources. We elicited judgements from participants on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using an interactive structured elicitation protocol, and we conducted 29 new high-powered replications. After interacting with their peers, participant groups with lower task expertise (‘beginners’) updated their estimates and confidence in their judgements significantly more than groups with greater task expertise (‘experienced’). For experienced individuals, the average accuracy was 0.57 (95% CI: [0.53, 0.61]) after interaction, and they correctly classified 61% of claims; beginners’ average accuracy was 0.58 (95% CI: [0.54, 0.62]), correctly classifying 69% of claims. The difference in accuracy between groups was not statistically significant and their judgements on the full set of claims were correlated (r(98) = 0.48, P < 0.001). These results suggest that both beginners and more-experienced participants using a structured process have some ability to make better-than-chance predictions about the reliability of ‘fast science’ under conditions of high uncertainty. However, given the importance of such assessments for making evidence-based critical decisions in a crisis, more research is required to understand who the right experts in forecasting replicability are and how their judgements ought to be elicited.
- 20 December 2024
- Published by Springer Nature in Nature Human Behaviour
- Vol. 9 (2) , 287-304
Open AccessPredicting the replicability of social and behavioural science claims in COVID-19 preprints
Replications are important for assessing the reliability of published findings. However, they are costly, and it is infeasible to replicate everything. Accurate, fast, lower-cost alternatives such as eliciting predictions could accelerate assessment for rapid policy implementation in a crisis and help guide a more efficient allocation of scarce replication resources. We elicited judgements from participants on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using an interactive structured elicitation protocol, and we conducted 29 new high-powered replications. After interacting with their peers, participant groups with lower task expertise (‘beginners’) updated their estimates and confidence in their judgements significantly more than groups with greater task expertise (‘experienced’). For experienced individuals, the average accuracy was 0.57 (95% CI: [0.53, 0.61]) after interaction, and they correctly classified 61% of claims; beginners’ average accuracy was 0.58 (95% CI: [0.54, 0.62]), correctly classifying 69% of claims. The difference in accuracy between groups was not statistically significant and their judgements on the full set of claims were correlated (r(98) = 0.48, P < 0.001). These results suggest that both beginners and more-experienced participants using a structured process have some ability to make better-than-chance predictions about the reliability of ‘fast science’ under conditions of high uncertainty. However, given the importance of such assessments for making evidence-based critical decisions in a crisis, more research is required to understand who the right experts in forecasting replicability are and how their judgements ought to be elicited.
- Open Access
- 19 November 2024
- Published by Springer Nature in Nature Human Behaviour
- Vol. 9 (2) , 316-330
Examining the replicability of online experiments selected by a decision market
Here we test the feasibility of using decision markets to select studies for replication and provide evidence about the replicability of online experiments. Social scientists (n = 162) traded on the outcome of close replications of 41 systematically selected MTurk social science experiments published in PNAS 2015–2018, knowing that the 12 studies with the lowest and the 12 with the highest final market prices would be selected for replication, along with 2 randomly selected studies. The replication rate, based on the statistical significance indicator, was 83% for the top-12 and 33% for the bottom-12 group. Overall, 54% of the studies were successfully replicated, with replication effect size estimates averaging 45% of the original effect size estimates. The replication rate varied between 54% and 62% for alternative replication indicators. The observed replicability of MTurk experiments is comparable to that of previous systematic replication projects involving laboratory experiments.
- 19 November 2024
- Published by Springer Nature in Nature Human Behaviour
- Vol. 9 (2) , 316-330
Open AccessExamining the replicability of online experiments selected by a decision market
Here we test the feasibility of using decision markets to select studies for replication and provide evidence about the replicability of online experiments. Social scientists (n = 162) traded on the outcome of close replications of 41 systematically selected MTurk social science experiments published in PNAS 2015–2018, knowing that the 12 studies with the lowest and the 12 with the highest final market prices would be selected for replication, along with 2 randomly selected studies. The replication rate, based on the statistical significance indicator, was 83% for the top-12 and 33% for the bottom-12 group. Overall, 54% of the studies were successfully replicated, with replication effect size estimates averaging 45% of the original effect size estimates. The replication rate varied between 54% and 62% for alternative replication indicators. The observed replicability of MTurk experiments is comparable to that of previous systematic replication projects involving laboratory experiments.
- Open Access
- 25 September 2024
- Published by The Royal Society in Royal Society Open Science
- Vol. 11 (9) , 240682
Can large language models help predict results from a complex behavioural science study?
We tested whether large language models (LLMs) can help predict results from a complex behavioural science experiment. In study 1, we investigated the performance of the widely used LLMs GPT-3.5 and GPT-4 in forecasting the empirical findings of a large-scale experimental study of emotions, gender, and social perceptions. We found that GPT-4, but not GPT-3.5, matched the performance of a cohort of 119 human experts, with correlations of 0.89 (GPT-4), 0.07 (GPT-3.5) and 0.87 (human experts) between aggregated forecasts and realized effect sizes. In study 2, providing participants from a university subject pool the opportunity to query a GPT-4 powered chatbot significantly increased the accuracy of their forecasts. Results indicate promise for artificial intelligence (AI) to help anticipate—at scale and minimal cost—which claims about human behaviour will find empirical support and which ones will not. Our discussion focuses on avenues for human–AI collaboration in science.
- 25 September 2024
- Published by The Royal Society in Royal Society Open Science
- Vol. 11 (9) , 240682
Open AccessCan large language models help predict results from a complex behavioural science study?
We tested whether large language models (LLMs) can help predict results from a complex behavioural science experiment. In study 1, we investigated the performance of the widely used LLMs GPT-3.5 and GPT-4 in forecasting the empirical findings of a large-scale experimental study of emotions, gender, and social perceptions. We found that GPT-4, but not GPT-3.5, matched the performance of a cohort of 119 human experts, with correlations of 0.89 (GPT-4), 0.07 (GPT-3.5) and 0.87 (human experts) between aggregated forecasts and realized effect sizes. In study 2, providing participants from a university subject pool the opportunity to query a GPT-4 powered chatbot significantly increased the accuracy of their forecasts. Results indicate promise for artificial intelligence (AI) to help anticipate—at scale and minimal cost—which claims about human behaviour will find empirical support and which ones will not. Our discussion focuses on avenues for human–AI collaboration in science.
- Top Cited PapersOpen Access
- 10 November 2023
- Published by Elsevier in Organizational Behavior and Human Decision Processes
- Vol. 179
On the trajectory of discrimination: A meta-analysis and forecasting survey capturing 44 years of field experiments on gender and hiring decisions
- 10 November 2023
- Published by Elsevier in Organizational Behavior and Human Decision Processes
- Vol. 179
Top Cited PapersOpen AccessOn the trajectory of discrimination: A meta-analysis and forecasting survey capturing 44 years of field experiments on gender and hiring decisions
- Open Access
- 15 February 2023
- Published by Center for Open Science in MetaArXiv
Predicting the replicability of social and behavioural science claims in COVID-19 preprints
Replications are important for assessing the reliability of published findings. However, they are costly, and it is infeasible to replicate everything. Accurate, fast, lower-cost alternatives such as eliciting predictions could accelerate assessment for rapid policy implementation in a crisis and help guide a more efficient allocation of scarce replication resources. We elicited judgments from participants on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using an interactive structured elicitation protocol, and we conducted 29 new high-powered replications. After interacting with their peers, participant groups with lower task expertise (‘beginners’) updated their estimates and confidence in their judgements significantly more than groups with greater task expertise (‘experienced’). For experienced individuals, the average accuracy was 0.57 (95% CI: [0.53, 0.61]) after interaction, and they correctly classified 61% of claims; beginners’ average accuracy was 0.58 (95% CI: [0.54, 0.62]), correctly classifying 69% of claims. The difference in accuracy between groups was not statistically significant, and their judgments on the full set of claims were correlated (r(98) = 0.48, p<0.001). These results suggest that both beginners and more experienced participants using a structured process have some ability to make better-than-chance predictions about the reliability of ‘fast science’ under conditions of high uncertainty. However, given the importance of such assessments for making evidence-based critical decisions in a crisis, more research is required to understand who the right experts in forecasting replicability are and how their judgements ought to be elicited.
- 15 February 2023
- Published by Center for Open Science in MetaArXiv
Open AccessPredicting the replicability of social and behavioural science claims in COVID-19 preprints
Replications are important for assessing the reliability of published findings. However, they are costly, and it is infeasible to replicate everything. Accurate, fast, lower-cost alternatives such as eliciting predictions could accelerate assessment for rapid policy implementation in a crisis and help guide a more efficient allocation of scarce replication resources. We elicited judgments from participants on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using an interactive structured elicitation protocol, and we conducted 29 new high-powered replications. After interacting with their peers, participant groups with lower task expertise (‘beginners’) updated their estimates and confidence in their judgements significantly more than groups with greater task expertise (‘experienced’). For experienced individuals, the average accuracy was 0.57 (95% CI: [0.53, 0.61]) after interaction, and they correctly classified 61% of claims; beginners’ average accuracy was 0.58 (95% CI: [0.54, 0.62]), correctly classifying 69% of claims. The difference in accuracy between groups was not statistically significant, and their judgments on the full set of claims were correlated (r(98) = 0.48, p<0.001). These results suggest that both beginners and more experienced participants using a structured process have some ability to make better-than-chance predictions about the reliability of ‘fast science’ under conditions of high uncertainty. However, given the importance of such assessments for making evidence-based critical decisions in a crisis, more research is required to understand who the right experts in forecasting replicability are and how their judgements ought to be elicited.
- Open Access
Can laypeople predict the replicability of social science studies without expert intervention: an exploratory study
The low replication rate of published studies has long concerned the social science community, making understanding the replicability a critical problem. Several studies have shown that relevant research communities can make predictions about the replicability of individual studies with above-chance accuracy. Follow-up work further indicates that laypeople can also achieve above-chance accuracy in predicting replicability when experts interpret the studies into short descriptions that are more accessible for laypeople. The involvement of scarce expert resources may make these methods expensive from financial and time perspectives. In this work, we explored whether laypeople can predict the replicability of social science studies without expert intervention. We presented laypeople with raw materials truncated from published social science papers and elicited their answers to questions related to the paper. Our results suggested that laypeople were engaged in this technical task, providing reasonable and self-contained answers. The majority of them also demonstrated a good understanding of the material. However, the solicited information had limited predictive power on the actual replication outcomes. We further discuss several lessons we learned compared to the approach with expert intervention to inspire future works.
Open AccessCan laypeople predict the replicability of social science studies without expert intervention: an exploratory study
The low replication rate of published studies has long concerned the social science community, making understanding the replicability a critical problem. Several studies have shown that relevant research communities can make predictions about the replicability of individual studies with above-chance accuracy. Follow-up work further indicates that laypeople can also achieve above-chance accuracy in predicting replicability when experts interpret the studies into short descriptions that are more accessible for laypeople. The involvement of scarce expert resources may make these methods expensive from financial and time perspectives. In this work, we explored whether laypeople can predict the replicability of social science studies without expert intervention. We presented laypeople with raw materials truncated from published social science papers and elicited their answers to questions related to the paper. Our results suggested that laypeople were engaged in this technical task, providing reasonable and self-contained answers. The majority of them also demonstrated a good understanding of the material. However, the solicited information had limited predictive power on the actual replication outcomes. We further discuss several lessons we learned compared to the approach with expert intervention to inspire future works.