Mark A. Burgman

This paper reports two approaches to forecasting replicability for a corpus of 3000 published social science papers, using large-scale human assessments. Replication Markets used incentivized surveys and prediction markets where participants traded assets linked to replication outcomes and market prices were interpreted as replication predictions. The repliCATS project used a structured group deliberation protocol, including interactive discussion, and mathematically aggregated forecasts to generate replication predictions. Accuracy for both approaches was validated against a subset of (n=37) independent, high-power replication studies. The predictive accuracy (median [range]) achieved for AUC by Replication Markets was 0.76 [0.72-0.82], and 0.76 [0.68-0.81] by repliCATS. Replication Markets achieved a classification accuracy of 73% [68-76%] and repliCATS achieved 68% [59-78%]. These results place the performance of both teams within the accuracy range achieved in prior replication forecasting studies. We conclude that informative forecasts can be elicited by both methods, but there are trade-offs between scale and accuracy.

The repliCATS-SCORE dataset consists of five human elicitation studies predicting the replicability of published research in the social and behavioural sciences, conducted by the repliCATS project (Collaborative Assessments for Trustworthy Science) as part of DARPA’s Systematizing Confidence in Open Research and Evidence or SCORE program (2019-2022). The human elicitations were conducted using a structured group deliberation technique – the IDEA protocol – for a total of 4000 published research claims and an additional 25 claim pilot study. The dataset for each study includes participant-level judgements across two elicitation rounds, mathematically aggregated group judgements or “confidence scores” calculated using a suite of 38 aggregation methods, as well as participant free-text reasoning. The repliCATS-SCORE dataset can be re-used in support of studies interrogating scholarly judgements of replicability, plausibility and overall credibility, post-publication peer review and critical evaluation training, developing novel mathematical aggregation models that forecast replication outcomes, and comparative studies.

This paper reports two approaches to forecasting replicability for a corpus of 3000 published social science papers, using large-scale human assessments. Replication Markets used incentivized surveys and prediction markets where participants traded assets linked to replication outcomes and market prices were interpreted as replication predictions. The repliCATS project used a structured group deliberation protocol, including interactive discussion, and mathematically aggregated forecasts to generate replication predictions. Accuracy for both approaches was validated against a subset of (n=37) independent, high-power replication studies. The predictive accuracy (median [range]) achieved for AUC by Replication Markets was 0.76 [0.72-0.82], and 0.76 [0.68-0.81] by repliCATS. Replication Markets achieved a classification accuracy of 73% [68-76%] and repliCATS achieved 68% [59-78%]. These results place the performance of both teams within the accuracy range achieved in prior replication forecasting studies. We conclude that informative forecasts can be elicited by both methods, but there are trade-offs between scale and accuracy.

Antarctica, long considered an environmental sanctuary, now confronts accelerating, complex and inter-related conservation challenges. The vast size and remote location of the continent introduce substantial uncertainty in understanding and predicting these threats. Here, using strategic foresight techniques, we synthesized insights from a global horizon scan with 131 experts from 42 countries. We identified ten emerging conservation challenges across six thematic categories. Key issues included extreme precipitation, emerging animal pathogens, human pandemics, security threats, reduced cooperation among Antarctic Treaty parties and potential agricultural expansion. Several of these challenges stem from persistent underlying drivers, revealing how longstanding processes are giving rise to new and increasingly acute conservation concerns. Others, driven by global disruptions, have no historical precedent in the region but increasingly constrain decision-making and international coordination. This horizon scan reveals substantial limitations in the ability of the Antarctic Treaty system to address these challenges, underscoring the need to reassess existing governance mechanisms to protect the unique ecosystems of Antarctica and its vital role in the global climate system.

Demand is increasing among investors to create portfolios that encourage positive outcomes for biological diversity. The evolution of investment strategies for transitions to zero carbon over the last two decades provides insights that will assist in shaping strategies for biodiversity‐positive investments. Many emerging approaches to capture company impact and dependence on biodiversity focus on nature‐related threats to an organisation by assessing ecosystem integrity. Other approaches focus on minimising an organisation's contribution to risks of species extinction by using data sets such as the IUCN's Red List of Threatened Species. However, while these approaches are useful for assessing threats to and from biodiversity for individual companies, to be effective for investment portfolio construction, metrics need to be comparable across companies in an investment universe. Many of the threat assessments that could link corporate activities to impacts are incomplete and omit critical information. If the investment community focuses on biodiversity without sufficient forethought, there is a risk of entrenching metrics with significant flaws. In this paper, we suggest that interim approaches are needed to support investors in understanding the approaches being taken by potential investee companies. To that end, we present and discuss a disclosure‐based Biodiversity Management Quality and a classification‐based Biodiversity Revenues metric for biodiversity‐related investing.

This paper explores judgements about the replicability of social and behavioural sciences research and what drives those judgements. Using a mixed methods approach, it draws on qualitative and quantitative data elicited from groups using a structured approach called the IDEA protocol (‘investigate’, ‘discuss’, ‘estimate’ and ‘aggregate’). Five groups of five people with relevant domain expertise evaluated 25 research claims that were subject to at least one replication study. Participants assessed the probability that each of the 25 research claims would replicate (i.e. that a replication study would find a statistically significant result in the same direction as the original study) and described the reasoning behind those judgements. We quantitatively analysed possible correlates of predictive accuracy, including self-rated expertise and updating of judgements after feedback and discussion. We qualitatively analysed the reasoning data to explore the cues, heuristics and patterns of reasoning used by participants. Participants achieved 84% classification accuracy in predicting replicability. Those who engaged in a greater breadth of reasoning provided more accurate replicability judgements. Some reasons were more commonly invoked by more accurate participants, such as ‘effect size’ and ‘reputation’ (e.g. of the field of research). There was also some evidence of a relationship between statistical literacy and accuracy.

Having collaborated with the community to decide upon the objectives, identified the major threats to be addressed, considered the possible options and assessed the evidence, the next stage is to decide what to do. Many decisions can be decided easily, as either obvious or trivial, thus requiring no additional assessment of different stages of the process. The harder decisions then require assessing the likely consequences of options and determining the preferred trade off. There is a range of approaches for making each of these stages more rigorous, reducing the likelihood of making inefficient decisions. These approaches are described with an account of the situations under which is most appropriate to address the conservation problem.

Biodiversity conservation decisions are difficult, especially when they involve differing values, complex multidimensional objectives, scarce resources, urgency, and considerable uncertainty. Decision science embodies a theory about how to make difficult decisions and an extensive array of frameworks and tools that make that theory practical. We sought to improve conceptual clarity and practical application of decision science to help decision makers apply decision science to conservation problems. We addressed barriers to the uptake of decision science, including a lack of training and awareness of decision science; confusion over common terminology and which tools and frameworks to apply; and the mistaken impression that applying decision science must be time consuming, expensive, and complex. To aid in navigating the extensive and disparate decision science literature, we clarify meaning of common terms: decision science , decision theory , decision analysis , structured decision‐making , and decision‐support tools . Applying decision science does not have to be complex or time consuming; rather, it begins with knowing how to think through the components of a decision utilizing decision analysis (i.e., define the problem, elicit objectives, develop alternatives, estimate consequences, and perform trade‐offs). This is best achieved by applying a rapid‐prototyping approach. At each step, decision‐support tools can provide additional insight and clarity, whereas decision‐support frameworks (e.g., priority threat management and systematic conservation planning) can aid navigation of multiple steps of a decision analysis for particular contexts. We summarize key decision‐support frameworks and tools and describe to which step of a decision analysis, and to which contexts, each is most useful to apply. Our introduction to decision science will aid in contextualizing current approaches and new developments, and help decision makers begin to apply decision science to conservation problems.

Conservation science deals with crises and supports policy interventions devised to mitigate highly uncertain threats that pose irreversible harm. When conventional policy tools, such as quantitative risk assessments, are insufficient, the precautionary principle provides a practical framework and range of robust heuristics. Yet, precaution is often resisted in many policy arenas, especially those involving powerful self‐interests, and this resistance is compounded by structures of privilege and competitive individualism in science. We describe key drivers and effects of such resistance in conservation science. These include a loss of rigor under uncertainty, an erosion of crisis response capabilities, and a further reinforcement of privileged interests in conservation politics. We recommend open acknowledgement of the pressures exerted by power inside science; greater recognition for the value of the precautionary principle under uncertainty; deliberate measures to resist competitive individualism; support for blind review, open science, and data sharing; and a shift from hierarchical multidisciplinarity toward more egalitarian transdisciplinarity to accelerate advances in conservation science. Article impact statement : Precautionary principle, privilege structures among disciplines, and culture of individualism link to effective conservation policy making.