Nicki Lisa Cole

Many concerns about the quality and integrity of the scientific literature have been raised in recent years. These concerns span a range of issues, including questionable research practices, defici...

This paper presents a three-year longitudinal autoethnographic study of the TIER2 project, an international, interdisciplinary consortium committed to “radical reproducibility” through open research practices. We investigated how epistemic diversity, disciplinary norms, and institutional cultures shape reproducibility in practice. Data include periodic consortium-wide surveys, quarterly reproducibility diaries by five diarists, and fieldnotes from General Assembly discussions; these were coded inductively. We found that initial disciplinary differences in defining reproducibility evolved to a nuanced appreciation of its complexity. Participants developed new competencies in open research practices and experienced insights regarding early planning and collaborative transparency. Enablers of reproducibility include the Open Science Framework, co-developed tools, containerization, and “slow science” approaches, strong role modelling by senior researchers and improved documentation. Challenges include fear of exposing imperfect work, unequal engagement across career stages, and substantial time and resource demands. Epistemic tensions emerged around the applicability of reproducibility to qualitative research and the risk of standardisation undermining diversity. Despite these, participants reported high professional satisfaction and intellectual growth. The study demonstrates that “radical reproducibility” is not merely a technical but a cultural and epistemic project that fosters collective learning, skill development and innovation, and requires systemic and institutional support.

Reproducibility of research is a hotly debated topic, including aspects like causes and consequences of low levels of reproducibility. While some research fields have led the way and introduced various reproducibility practices and procedures, the call for efforts to ‘improve’ reproducibility in research has not come without criticisms. The current study uses a future studies methodology to gather perceptions of developments in the research ecosystem related to reproducibility issues. It draws on input from representatives of four main stakeholder categories: scholarly publishers, funding agencies, qualitative social scientists and machine learning researchers. Particularly, it discusses the enablers and barriers that members of these stakeholder communities foresee on the road towards a research ecosystem that is more conducive to reproducibility. The study finds that enablers and barriers can be categorised into five main clusters. The factors most prominently mentioned as potentially supporting or hindering a desired future are located within research culture, including norms, values and shared definitions; and in the infrastructure required to engage in reproducibility practices, including repositories, support staff, and digital infrastructure. Three other clusters of factors put forth by participants relate to policy efforts required to incentivise reproducibility practices; training and education to empower researchers and support staff to engage in reproducibility practices; and the financial resources required to facilitate the transition towards a desired future and to specifically fund replication studies. This manuscript also identifies several tensions between enablers and barriers perceived by diverse stakeholders and concludes with recommendations for addressing these.

Reproducibility of research is a hotly debated topic, including aspects like causes and consequences of low levels of reproducibility. While some research fields have led the way and introduced various reproducibility practices and procedures, the call for efforts to ‘improve’ reproducibility in research has not come without criticisms. The current study uses a future studies methodology to gather perceptions of developments in the research ecosystem related to reproducibility issues. It draws on input from representatives of four main stakeholder categories: scholarly publishers, funding agencies, qualitative social scientists and machine learning researchers. Particularly, it discusses the enablers and barriers that members of these stakeholder communities foresee on the road towards a research ecosystem that is more conducive to reproducibility.The study finds that enablers and barriers can be categorised into five main clusters. The factors most prominently mentioned as potentially supporting or hindering a desired future are located within research culture, including norms, values and shared definitions; and in the infrastructure required to engage in reproducibility practices, including repositories, support staff, and digital infrastructure. Three other clusters of factors put forth by participants relate to policy efforts required to incentivise reproducibility practices; training and education to empower researchers and support staff to engage in reproducibility practices; and the financial resources required to facilitate the transition towards a desired future and to specifically fund replication studies. This manuscript also identifies several tensions between enablers and barriers perceived by diverse stakeholders and concludes with recommendations for addressing these.

Reproducibility of research is a hotly debated topic, including aspects like causes and consequences of low levels of reproducibility. While some research fields have led the way and introduced various reproducibility practices and procedures, the call for efforts to ‘improve’ reproducibility in research has not come without criticisms. The current study uses a future studies methodology to gather perceptions of developments in the research ecosystem related to reproducibility issues. It draws on input from representatives of four main stakeholder categories: scholarly publishers, funding agencies, qualitative social scientists and machine learning researchers. Particularly, it discusses the enablers and barriers that members of these stakeholder communities foresee on the road towards a research ecosystem that is more conducive to reproducibility.The study finds that enablers and barriers can be categorised into five main clusters. The factors most prominently mentioned as potentially supporting or hindering a desired future are located within research culture, including norms, values and shared definitions; and in the infrastructure required to engage in reproducibility practices, including repositories, support staff, and digital infrastructure. Three other clusters of factors put forth by participants relate to policy efforts required to incentivise reproducibility practices; training and education to empower researchers and support staff to engage in reproducibility practices; and the financial resources required to facilitate the transition towards a desired future and to specifically fund replication studies. This manuscript also identifies several tensions between enablers and barriers perceived by diverse stakeholders and concludes with recommendations for addressing these.

The integrative review presented here examines how reproducibility and replicability are conceptualized and discussed in relation to qualitative research, and which factors and practices enable or undermine them. Both peer-reviewed and grey English-language literature that address reproducibility and/or Open Science in relation to qualitative research were eligible for inclusion. Initial searches were conducted in Scopus, Web of Science, Dimensions, PubMed, APA PsychInfo, and JSTOR, and followed with snowball sampling from included literature. Studies were screened and both quantitative and qualitative data were extracted using the SyRF online platform, with 248 papers included. We found that conceptualizations that stem from quantitative standpoints are overwhelmingly framed as inappropriate practices and epistemic criteria for (most) qualitative research. When conceptualized in alternative ways that are adapted to the epistemic conditions, aims and practices of qualitative research, they can be both applicable and appropriate. Key barriers include the ontological and epistemological misalignment of reproducibility, replicability and Open Science and qualitative research, and ethical and practical concerns surrounding data sharing and reuse. Key enablers include practices that respond to ethical and practical concerns around data sharing and reuse (anonymization, ethical consent practices, context documentation, and ethical access management), adapting expectations and norms of openness, and established qualitative practices including documentation, reflexivity, and considering positionality. We conclude that reproducibility, replicability and Open Science practices must be adapted to the aims and epistemic conditions of qualitative research for them to be applicable and feasible, and that they will not always be both for all qualitative research.

The integrative review presented here examines how reproducibility and replicability are conceptualized and discussed in relation to qualitative research, and which factors and practices enable or undermine them. Both peer-reviewed and grey English-language literature that address reproducibility and/or Open Science in relation to qualitative research were eligible for inclusion. Initial searches were conducted in Scopus, Web of Science, Dimensions, PubMed, APA PsychInfo, and JSTOR, and followed with snowball sampling from included literature. Studies were screened and both quantitative and qualitative data were extracted using the SyRF online platform, with 248 papers included. We found that conceptualizations that stem from quantitative standpoints are overwhelmingly framed as inappropriate practices and epistemic criteria for (most) qualitative research. When conceptualized in alternative ways that are adapted to the epistemic conditions, aims and practices of qualitative research, they can be both applicable and appropriate. Key barriers include the ontological and epistemological misalignment of reproducibility, replicability and Open Science and qualitative research, and ethical and practical concerns surrounding data sharing and reuse. Key enablers include practices that respond to ethical and practical concerns around data sharing and reuse (anonymization, ethical consent practices, context documentation, and ethical access management), adapting expectations and norms of openness, and established qualitative practices including documentation, reflexivity, and considering positionality. We conclude that reproducibility, replicability and Open Science practices must be adapted to the aims and epistemic conditions of qualitative research for them to be applicable and feasible, and that they will not always be both for all qualitative research.

The UNESCO Recommendations for Open Science define it as an inclusive framework that integrates movements and practices aimed at making multilingual scientific knowledge openly accessible, reusable, and collaborative for the benefit of science and society. This approach emphasizes transparency, collaboration, rigor, and accessibility, principles that align closely with qualitative research methodologies. In the health and social sciences, qualitative methods explore the meanings, experiences, and perspectives of individuals and groups. Open qualitative research encompasses diverse practices, including transparency in design and analysis, enabling meaningful scrutiny, and fostering collaboration through stakeholder engagement and co-production. The application of openness in qualitative research varies based on the context and specific project needs.

IntroductionOpen Science (OS) is increasingly mainstream, with wide adoption of practices like Open Access and FAIR data. While systems and data sources for monitoring the uptake of these practices are now relatively mature, research to establish real-world, long-term impacts is largely scattered and patchy. In this talk, I (on behalf of the PathOS project team) will present work to scope and synthesise current evidence of: Academic impact: E.g., OS contributions to research efficiency, quality, collaboration, and equity. Societal impact: E.g., effects on policy, governance, education, climate, public trust, and health. Economic impact: E.g., influence on industry productivity, innovation, and cost efficiency. We used the PRISMA-SCR methodology (Tricco et al. 2018) to systematically scope literature from 2000-2022. Starting from a total of more than 30,000 initially retrieved records, the study team identified, selected, analysed, and synthesised peer-reviewed literature from academic databases and further “grey literature”/preprints from the Web. Full results for two studies (academic and societal impact) are already available online (Cole et al. 2024; Klebel et al. 2024). The third, on economic impact, is in preparation and will soon be made available as a preprint and submitted for journal publication. Results415 studies demonstrated academic impact, 196 societal impact, and 70 economic impact. Most academic studies focus on Open Access, showing increased citation rates but raising equity concerns due to the article processing charge (APC) model. Citizen Science improves data collection and education but has varying levels of data quality. Open/FAIR data leads to reuse benefits and better reproducibility. Societal impact studies highlight Citizen Science’s role in education, climate awareness, and policy engagement, though Open Access’s societal impact remains limited. Economic studies suggest theoretical benefits like cost savings and innovation, but empirical data is scarce. Discussion and conclusionThis talk will detail and reflect upon the these findings, including the uneven distribution of research on OS impacts, difficulties in assessing OS impact and establishing causality, the tendency for research to focus on areas where data is evenly available rather than where evidence is most needed, prior focus on uptake rather than impact, and the implications of all this for our understanding OS impact. Finally we will look ahead to PathOS’ future work on OS impact indicators, methods for cost-benefit analysis, and specific case-study investigations to deepen our understanding of OS’s influence on research efficiency, quality, equity, societal engagement, policy-making, and economic outcomes. View the presentation in this video recording.

Open Science (OS) aims, in part, to drive greater societal impact of academic research. Government, funder and institutional policies state that it should further democratize research and increase learning and awareness, evidence-based policy-making, the relevance of research to society's problems, and public trust in research. Yet, measuring the societal impact of OS has proven challenging and synthesized evidence of it is lacking. This study fills this gap by systematically scoping the existing evidence of societal impact driven by OS and its various aspects, including Citizen Science (CS), Open Access (OA), Open/FAIR Data (OFD), Open Code/Software and others. Using the PRISMA Extension for Scoping Reviews and searches conducted in Web of Science, Scopus and relevant grey literature, we identified 196 studies that contain evidence of societal impact. The majority concern CS, with some focused on OA, and only a few addressing other aspects. Key areas of impact found are education and awareness, climate and environment, and social engagement. We found no literature documenting evidence of the societal impact of OFD and limited evidence of societal impact in terms of policy, health, and trust in academic research. Our findings demonstrate a critical need for additional evidence and suggest practical and policy implications.