Stephen Pinfield

BackgroundPeer review-based funding evaluation systems are under increasing pressure. To help alleviate this, innovative approaches like Distributed Peer Review (DPR) – where applicants review proposals for the same funding call – are being explored. The Volkswagen Foundation trialled DPR alongside a conventional panel review process for an interdisciplinary call. Evaluation of this trial provides insights into the dynamics of (distributed) peer review. MethodsAll proposals (N=140) underwent both DPR and the panel review process. In DPR, applicants (N= 323) reviewed four or five proposals, providing an overall score, scores for each funding criteria, and a comment. Proposals were ranked using a trimmed mean score. Expectation and feedback surveys were administered. The panel process involved shortlisting by VWS staff, review by panellists (N=8), and a final panel discussion. Semi-structured interviews with applicants, panellists, and VWS staff (N=20) explored perceptions and experiences of DPR, using thematic analysis. Quantitative analysis of review data and surveys utilised bootstrapping to assess decision stability and mixed effects models for consistency between reviewers. Qualitative and quantitative results were integrated in a convergent parallel design.ResultsDPR recommended ten proposals for funding, while the panel recommended eleven; three overlapped. Funding criteria strongly predicted overall score; ‘Originality’ was given most weight by both sets of reviewers. Stability of DPR funding decisions was modest; across 500 bootstrapped samples a mean of 5.49 out of ten previously funded proposals would still be funded. This may be explained by high inter-reviewer variability: more variance was attributable to between-reviewer differences (20.0%) than between-proposal differences (8.9%). Advantages of DPR included increased efficiency, faster decision times, detailed feedback to applicants, and democratisation of funding evaluation. Noted disadvantages were burden on applicants, absence of discussion, and perceived lack of reviewer expertise. Funders implementing DPR must consider the high initial effort, potential for ‘gaming’, and reduced direct influence. ConclusionsThis trial demonstrates the feasibility of DPR for an interdisciplinary funding programme. Implementing DPR requires funders to accept trade-offs and that DPR may not produce the same outcomes as existing systems. Building on established peer review principles, it may offer benefits, including shortening time to funding decision, and has the potential to contribute to the democratisation of funding evaluation.

Background Discourses of ‘Open Science’ have been criticised for privileging the quantitative and positivist methodologies of science, technology, engineering and mathematics (STEM) research at the expense of qualitative, interpretive, critical-theoretical, practice-based, and other forms of research more common in the arts, humanities and social sciences (AHSS). Methods In this article, which emerged from a work component of the MORPHSS (Materialising Open Research Practices in the Humanities and Social Sciences) project, we document the process, outcomes and practical implications of work to develop a catalogue of open research practices in these disciplines, which proceeded via a wide-ranging literature review followed by targeted desk research and a conceptual mapping exercise. Results Key findings include the fact that open research practices in AHSS are diverse, extending beyond the suite of practices emphasised within dominant accounts of Open Science. We identify among these practices a range of forms of openness including those focused on mobilising the involvement and expert knowledge of diverse participants and communities. Conclusions Presenting a typology of forms of openness in AHSS that is consistent with the epistemic logics of these disciplines, we conclude that openness in AHSS is highly situated and context-dependent, as well as resisting quantification and binary measurement. Drawing on these conclusions, we offer a series of recommendations for institutions, open research monitoring initiatives, funders, publishers, learned societies and researchers to enhance the inclusivity of their policy and practice around openness.

Expectations around qualitative data sharing have a vexed history. While some assert their importance, including in facilitating the impact of rich data beyond the confines of an individual research project, others highlight the barriers around sharing data that is often personal, sensitive, and highly situated, indicating a need for ongoing research on how these challenges are perceived and negotiated within specific methodological contexts. Responding to a scarcity of both research on ethnographers’ data sharing practices and attention to qualitative sharing in the context of mixed methods, this article details the findings from a series of 22 interviews with ethnographic and mixed methods researchers regarding their decision-making processes around whether and how to share qualitative data. We find that interviewees’ consideration of the rationale for and benefits of qualitative data sharing goes far beyond a focus on reproducibility or verifiability to embrace a broad range of factors. Among both mixed methods and ethnographic researchers, data is perceived as both a rigorous research output carrying profound evidentiary weight and a product of a co-created, contextually-situated endeavour, a finding that is reflected in the variety of factors that inform decision-making around whether and how it can be shared. This necessitates, we suggest, care-full, situated negotiation between the needs and interests of diverse actors alongside pragmatic handling of logistical parameters. After detailing the moderating mechanisms used to achieve such negotiation in instances where data sharing is possible, we conclude by highlighting the supportive environment within which such processes of care must in themselves take place.

BackgroundDistributed peer review (DPR) of grant proposals, where applicants review the proposals of others applying to the same funding call, offers advantages, including increased feedback to applicants. With the reviewer pool being determined by the applicants to a particular call, there are, however, concerns that reviewers may not possess the necessary expertise and experience to provide quality reviews and feedback. We analysed review comments from recent trials of DPR for funding evaluation at the Volkswagen Foundation to obtain insights into reviewing and the impact of reviewer characteristics on review content. MethodReview comments across two years (N=2950) were classified (content and sentiment) using existing, fine-tuned machine learning classifier models. Mean prevalence of content categories and sentiment was calculated, both overall and stratified by reviewer/proposal characteristics. ResultsReview comments were consistent with review scores and overall ranking of proposals. Minor differences in content and sentiment according to reviewer gender and research experience (years working in research) were detected. There was minimal variation according to funding peer review experience and disciplinary alignment between reviewer and proposal. ConclusionExisting classifier models can be used to analyse grant review comments and to obtain insights into reviewer consistency and adherence to funding criteria. Concerns about the influence of reviewer experience and expertise on reviews may be unfounded. Results from analyses of review comments can be used to audit and refine review processes.

BackgroundPeer review-based funding evaluation systems are under increasing pressure. To help alleviate this, innovative approaches like Distributed Peer Review (DPR) – where applicants review proposals for the same funding call – are being explored. The Volkswagen Foundation trialled DPR alongside conventional panel review for an interdisciplinary call. Evaluation of this trial provides insights into the dynamics of (distributed) peer review. MethodsAll proposals (N=140) underwent both DPR and panel review. In DPR, applicants (N= 323) reviewed four or five proposals, providing an overall score, scores for each funding criteria, and a comment. Proposals were ranked using a trimmed mean score. Expectation and feedback surveys were administered. Panel review involved shortlisting by VWS staff, panellist review (N=8), and a final panel discussion. Semi-structured interviews with applicants, panellists, and VWS staff (N=20) explored perceptions and experiences of DPR, using thematic analysis. Quantitative analysis of review data and surveys utilised bootstrapping to assess decision stability and mixed effects models for consistency between reviewers. Qualitative and quantitative results were integrated in a convergent parallel design. ResultsDPR recommended ten proposals for funding, while the panel recommended eleven; three overlapped. Funding criteria strongly predicted overall score; ‘Originality’ was given most weight by both sets of reviewers. Stability of DPR funding decisions was modest; across 500 bootstrapped samples a mean of 5.49 out of ten previously funded proposals would still be funded. This may be explained by high inter-reviewer variability: more variance was attributable to between-reviewer differences (20.04%) than between-proposal differences (8.87%). Advantages of DPR included increased efficiency, faster decision times, detailed feedback to applicants, and democratisation of funding evaluation. Noted disadvantages were burden on applicants, absence of discussion, and perceived lack of reviewer expertise. Funders implementing DPR must consider the high initial effort, potential for ‘gaming’, and reduced direct influence. ConclusionsThis trial demonstrates feasibility of DPR for an interdisciplinary funding programme. Implementing a DPR process requires funders to accept trade-offs, but can offer valuable benefits, including shortening time to funding decision. Building on established peer review principles, it has the potential to contribute to the democratisation of funding evaluation.

Background Discourses of ‘Open Science’ have been criticised for privileging the quantitative and positivist methodologies of science, technology, engineering and mathematics (STEM) research at the expense of qualitative, interpretive, critical-theoretical, practice-based, and other forms of research more common in the arts, humanities and social sciences (AHSS). Methods In this article, which emerged from a work component of the MORPHSS (Materialising Open Research Practices in the Humanities and Social Sciences) project, we document the process, outcomes and practical implications of work to develop a catalogue of open research practices in these disciplines, which proceeded via a wide-ranging literature review followed by targeted desk research and a conceptual mapping exercise. Results Key findings include the fact that open research practices in AHSS are diverse, extending beyond the suite of practices emphasised within dominant accounts of Open Science. We identify among these practices a range of forms of openness including those focused on mobilising the involvement and expert knowledge of diverse participants and communities. Conclusions Presenting a typology of forms of openness in AHSS that is consistent with the epistemic logics of these disciplines, we conclude that openness in AHSS is highly situated and context-dependent, as well as resisting quantification and binary measurement. Drawing on these conclusions, we offer a series of recommendations for institutions, open research monitoring initiatives, funders, publishers, learned societies and researchers to enhance the inclusivity of their policy and practice around openness.

As preprint servers proliferate across scientific fields, moderation has emerged as a lightly structured form of gatekeeping that precedes—or bypasses—peer review. This article examines moderation as a site of precautionary valuation and boundary-making in the early circulation of research. Drawing on interviews with 14 individuals responsible for screening and oversight across 13 preprint servers, we analyze both the evaluative logics that organize moderation and the ways moderators themselves understand and frame their work. We find that credibility is enacted through precaution-driven threshold judgments—centering harm, trust, and community expectations—which determine whether a submission can responsibly circulate as scholarship. While moderators often reject the label of “gatekeeper,” our findings show that their work performs a subtle but consequential filtering that shapes what and who enters the scholarly record—even as servers work to distance these decisions from the domain of peer review. By tracing how precautionary judgments and boundary-making structure decisions about what enters circulation at the preprint stage, we contribute to STS debates on valuation, peer review, and the governance of scholarly communication in open science.

BackgroundPeer review-based funding evaluation systems are under increasing pressure. To help alleviate this, innovative approaches like Distributed Peer Review (DPR) – where applicants review proposals for the same funding call – are being explored. The Volkswagen Foundation trialled DPR alongside conventional panel review for an interdisciplinary call. Evaluation of this trial provides insights into the dynamics of (distributed) peer review. MethodsAll proposals (N=140) underwent both DPR and panel review. In DPR, applicants (N= 323) reviewed four or five proposals, providing an overall score, scores for each funding criteria, and a comment. Proposals were ranked using a trimmed mean score. Expectation and feedback surveys were administered. Panel review involved shortlisting by VWS staff, panellist review (N=8), and a final panel discussion. Semi-structured interviews with applicants, panellists, and VWS staff (N=20) explored perceptions and experiences of DPR, using thematic analysis. Quantitative analysis of review data and surveys utilised bootstrapping to assess decision stability and mixed effects models for consistency between reviewers. Qualitative and quantitative results were integrated in a convergent parallel design. ResultsDPR recommended ten proposals for funding, while the panel recommended eleven; three overlapped. Funding criteria strongly predicted overall score; ‘Originality’ was given most weight by both sets of reviewers. Stability of DPR funding decisions was modest; across 500 bootstrapped samples a mean of 5.49 out of ten previously funded proposals would still be funded. This may be explained by high inter-reviewer variability: more variance was attributable to between-reviewer differences (20.04%) than between-proposal differences (8.87%). Advantages of DPR included increased efficiency, faster decision times, detailed feedback to applicants, and democratisation of funding evaluation. Noted disadvantages were burden on applicants, absence of discussion, and perceived lack of reviewer expertise. Funders implementing DPR must consider the high initial effort, potential for ‘gaming’, and reduced direct influence. ConclusionsThis trial demonstrates feasibility of DPR for an interdisciplinary funding programme. Implementing a DPR process requires funders to accept trade-offs, but can offer valuable benefits, including shortening time to funding decision. Building on established peer review principles, it has the potential to contribute to the democratisation of funding evaluation.

We analyse focus group discussions and free-text survey responses from a multi-stakeholder consultation conducted after the October 2023 publication of the proposal Towards Responsible Publishing by cOAlition S. The proposal calls for a systemic reform of scholarly communication by reducing barriers to knowledge dissemination, promoting early sharing of outputs through preprints, and shifting peer review to an open, post-publication model. We focus on how different stakeholder groups –such as researchers, infrastructure providers, academic institutions, and publishers –perceive obstacles to the large-scale, coordinated reform envisioned in the proposal. We interpret these accounts as articulations of collective action problems, shaped by entrenchment of many actors in existing academic reward systems and established commercial revenue models that make transitions toward a more economically sustainable scholarly communication system difficult, even where many actors see the principal need for change. This approach highlights the extent to which stakeholder perspectives align or conflict. It also underscores the performative nature of discourse about collective action problems in scholarly communication: by articulating challenges to reform, participants simultaneously construct, reinforce, or contest their own roles within the system, which directly influences their collective capacity to act.

Preprinting has become an increasingly important component of the scholarly communication system, facilitating rapid open dissemination of scientific knowledge. This study investigates the adoption of preprinting over time, focusing on how it varies across scientific disciplines and geographical regions. We analyzed bibliometric data on 4M preprints and 105M peer-reviewed outputs in the period 1991-2023. Peer-reviewed outputs were linked to preprints using data from Dimensions, OpenAlex, and Crossref, resulting in 2.2M peer-reviewed outputs linked to a preprint. Our findings indicate a strong growth in preprinting, with a nearly threefold increase in the number of preprints published between 2017 and 2022. The adoption of preprinting is highest in the physical and mathematical sciences, particularly among researchers in the Americas and Europe. In recent years, preprinting has also increased notably in the information and computing sciences and the life and medical sciences, driven primarily by researchers in North America and Western and Northern Europe. Preprinting remains relatively uncommon in the humanities and the social and behavioral sciences. Asia demonstrates low preprint adoption, with Eastern Asia showing a modest increase in recent years. Preprint adoption in specific disciplines varies significantly across regions, showing that preprint adoption is shaped by the interplay between disciplines and regions.