Teaching in the Garden of Forking Paths: Implementing Many-Analysts Designs in the Classroom

Authors:
  • KFKarolin Freitag
    Karolin Freitag
  • LKLaura Kiemes
    Laura Kiemes
  • LBLinda Biester
    Linda Biester
  • PBPaul Binder
    Paul Binder
  • BBBastian Buitkamp
    Bastian Buitkamp
  • LDLarissa Dyk
    Larissa Dyk
  • LELouisa Ehlich
    Louisa Ehlich
  • MLMariia Lesiv
    Mariia Lesiv
  • YPYannick Poliandri
    Yannick Poliandri
  • CSCelina Schneider
    Celina Schneider
  • ABAdrian Brenner
    Adrian Brenner
Preprint
Publisher website
Google Scholar
Abstract
The rise of many-analysts studies has underscored substantial variability in research outcomes when different teams independently analyze identical data sets and hypotheses. This paper demonstrates how the many-analysts framework can be effectively integrated into research methods education. We propose a pedagogical approach in which multiple students independently analyze the same research question using identical datasets in their term papers, followed by a meta-analysis of their collective results. This instructional method highlights critical methodological concepts, such as researcher degrees of freedom and epistemic humility. Moreover, this study applies the many-analysts approach specifically to causal inference using observational data with difference-in-differences analyses. The analyses conducted by students revealed a broad range of effect sizes and led to divergent conclusions, thus emphasizing the educational and methodological value of robustness checks involving multiple independent analysts. These findings illustrate how integrating a many-analysts framework in teaching can enhance students' methodological rigor and appreciation of empirical uncertainty.