Kaitlyn Hair

The emergence of various AI-enabled systematic review platforms has promised to transform evidence synthesis by providing more rapid, comprehensive and accurate reviews. However, independent validation of these platforms has been limited. Here, we set out to evaluate the performance of Elicit, SciSpace, and Nested Knowledge in each replicating three contemporary systematic reviews. We terminated our evaluation for SciSpace on the grounds of futility following critical data export issues. Mean sensitivity in identifying and including relevant publications was 33.3% for Elicit and 78.8% for Nested Knowledge. Mean percentage of overall correct annotations was 69.1% for Elicit and 58.6% for Nested Knowledge, but there was substantial variation between reviews and type of annotations. Agreement of risk of bias evaluations with the original reviews was low for both Elicit (mean κ = 0.04) and Nested Knowledge (mean κ = 0.18). At the time of our study, Elicit could not perform quantitative analysis. Nested Knowledge’s quantitative data extraction feature could not be applied to our reviews due to its system limitations. Elicit tended to draw more confident conclusions compared with the original reviews. In conclusion, Elicit is not currently suitable in replacing human reviewer efforts but could be used to supplement traditional search methods. Human-in-the-loop approaches implemented by Nested Knowledge may provide efficiency gains in screening, annotation and data extraction, but users should be mindful of system limitations, variation in performance across different tasks and review subject area, and consider if the available evaluation data is relevant to their use cases.

Systematic reviews of preclinical data are helpful in informing experimental design and translation, yet their own reproducibility has not been evaluated. We conducted a community-led Reproducibility Challenge to address this gap. Five international teams assessed reproducibility and identified factors affecting the replication success for four published reviews involving animal models of disease. Most teams successfully reproduced the original conclusions. However, we identified important quantitative and qualitative discrepancies in replicating individual steps of the systematic reviews. Differences arose in study selection, data processing and meta-analysis, and risk of bias assessments, due largely to incomplete or ambiguous reporting in the original reviews and to restricted access to underlying data and analysis code. Our findings suggest that preclinical systematic reviews may be more robust to replication than studies assessed in initiatives from other fields, although a lack of methodological transparency remains a significant barrier to exact replication. We provide recommendations, including refinements to reporting standards, to strengthen the transparency of preclinical evidence syntheses. We demonstrate that such collaborative challenges serve as a scalable model for community-driven engagement with reproducibility and the validation of systematic review findings. Wider adoption of reporting standards and open science practices is needed to ensure that preclinical systematic reviews provide a reliable foundation for translational research.

A key task in conducting systematic reviews is deduplicating the results from database searching. Deduplication using reference management software can be time-consuming and prone to error, while automated tools can be expensive and lack transparency. To support review teams, we evaluated eight deduplication tools: (1) The Automated Systematic Search Deduplicator (ASySD); (2) Covidence; (3) Deduklick; (4) EPPI-Reviewer; (5) PICO Portal; (6) Rayyan; (7) The Systematic Review Accelerator (SRA) Deduplicator: Focused; (8) The SRA Deduplicator: Relaxed. Five randomly selected Cochrane reviews had their searches rerun to create five gold standard sets. We compared the gold standard sets to the outputs of the eight deduplication tools and evaluated the results for: (1) unique records removed; (2) duplicate records retained; (3) time taken to deduplicate. Summed across all five reviews, the unique records removed in error ranged from 2 to 22. The three best tools were: (1) Rayyan; (2) Covidence; (3) SRA Deduplicator: Focused. The duplicate records retained in error ranged from 34 to 280, the three best tools were: (1) ASySD; (2) Rayyan; (3) EPPI-Reviewer. The time taken to deduplicate ranged from one minute to 20 hours and 34 minutes, the three fastest tools were: (1) SRA Deduplicator: Relaxed; (2) Deduklick; (3) Covidence. No tool performed so poorly that we don’t recommend using it. But, as all the tools had strengths and weaknesses, some are expensive while others require large amounts of manual checking time, we recommend review teams compare the tools across all three outcomes and choose the tool that best suits their needs.

Deduplicates bibliographic citations from multiple sources while preserving customizable metadata, supporting data-driven search strategy development and evidence synthesis reporting. Search results can be analyzed using plots and tables, and imported or exported in 'RIS' and 'CSV' formats. An interactive 'shiny' application is included for exploratory use.

Evidence synthesis findings hinge upon well-designed, effective search strategies. When developing these strategies, evidence synthesis teams make multiple decisions (e.g., selecting information sources, developing search string architecture, and picking supplementary search methods) that directly affect the breadth of discovered evidence and thus evidence synthesis outcomes. Despite the number of decisions required when developing search strategies, limited guidance exists to inform these decisions using a data-driven approach. To help address this gap, we developed CiteSource, an R package and accompanying Shiny application, that supports data-driven search strategy development and reporting. CiteSource allows users to assign and retain metadata across three custom fields: source, label , and string to indicate where the records were found, what method or string was used to find them, and whether they were included after screening. CiteSource allows users to visually map the overlap between sets of records, create data summaries of citation records, and export citation records with the newly assigned metadata. CiteSource’s analysis and visualization outputs can be harnessed for a variety of use cases, such as optimizing literature source selection, honing and understanding the effectiveness of search strings, and evaluating the impacts of literature sources and supplementary search methods. Overall, CiteSource provides a tool for evidence synthesizers to make informed data-driven decisions that boost the efficiency, rigor, and transparency of search strategies and associated reporting.

The causes of the reproducibility crisis include lack of standardization and transparency in scientific reporting. Checklists such as ARRIVE and CONSORT seek to improve transparency, but they are not always followed by authors and peer review often fails to identify missing items. To address these issues, there are several automated tools that have been designed to check different rigor criteria. We have conducted a broad comparison of 11 automated tools across 9 different rigor criteria from the ScreenIT group. We found some criteria, including detecting open data, where the combination of tools showed a clear winner, a tool which performed much better than other tools. In other cases, including detection of inclusion and exclusion criteria, the combination of tools exceeded the performance of any one tool. We also identified key areas where tool developers should focus their effort to make their tool maximally useful. We conclude with a set of insights and recommendations for stakeholders in the development of rigor and transparency detection tools. The code and data for the study is available at https://github.com/PeterEckmann1/tool-comparison .

The scholarly publishing system is adapting to many changes, including open access and open data mandates, artificial intelligence, and other new technologies. Members of the research and publishing communities are working to establish a more equitable, fair, and rigorous system that serves researchers' evolving needs. Early career researchers (ECRs) are drivers of change, and publishers may wonder why and how they should involve ECRs in shaping the future of scholarly publishing. We held a virtual unconference to explore this issue with publishers and ECRs who were working to improve publishing. Some participants sought to improve peer reviewer or editor performance, whereas others sought to improve the publishing system itself through iterative or transformative change. Strategies for collaborating with ECRs to shape the future of scholarly publishing included peer review programmes, editorial programmes, ECR‐led journals, ECR boards and committee representatives, and other ECR‐initiated activities. ECRs particularly wanted to see three things improved: (1) Sharing research outputs other than publications, (2) addressing technological limitations to create systems that meet the research community's needs and facilitate knowledge advancement, and (3) fostering diversity, equity, inclusion, and accessibility. We offer tips for publishers on how to collaborate with ECRs to enhance scholarly publishing, appeal to and learn from younger researchers, and better meet researchers' needs.

Publications reporting research conducted on human participants, such as clinical trials,commonly include a table summarising the relevant characteristics of the study participants.These tables are crucial to compare studies, assess generalisabillity and identify understudiedpopulations. In metaresearch and evidence syntheses, where the sample is instead comprisedof research papers, tables describing features such as the proportion of open access papers,publication language, or authors’ gender are uncommon. Collating these additionalcharacteristics manually would require substantial time and resources. Without thisinformation, however, our understanding of the generalisability of study findings is limited.Furthermore, researchers cannot compare sample characteristics across studies or assess theimpact of different sampling methods. This project aims to: 1) propose a standard samplecharacteristics table structure for studies performed on research papers, 2) determine which ofthese characteristics can be assessed by automated tools, and 3) create a web application thatallows researchers to quickly generate a sample characteristics table using existing automatedtools. We will share progress by updating this preprint as we complete these goals, allowingothers to provide feedback and contribute. We hope that this project will mainstream samplecharacteristics reporting for studies on research papers, stimulate automated tool developmentto fill gaps, and facilitate studies on sample characteristics and generalisability in metaresearch.

Publications reporting research conducted on human participants, such as clinical trials,commonly include a table summarising the relevant characteristics of the study participants.These tables are crucial to compare studies, assess generalisabillity and identify understudiedpopulations. In metaresearch and evidence syntheses, where the sample is instead comprisedof research papers, tables describing features such as the proportion of open access papers,publication language, or authors’ gender are uncommon. Collating these additionalcharacteristics manually would require substantial time and resources. Without thisinformation, however, our understanding of the generalisability of study findings is limited.Furthermore, researchers cannot compare sample characteristics across studies or assess theimpact of different sampling methods. This project aims to: 1) propose a standard samplecharacteristics table structure for studies performed on research papers, 2) determine which ofthese characteristics can be assessed by automated tools, and 3) create a web application thatallows researchers to quickly generate a sample characteristics table using existing automatedtools. We will share progress by updating this preprint as we complete these goals, allowingothers to provide feedback and contribute. We hope that this project will mainstream samplecharacteristics reporting for studies on research papers, stimulate automated tool developmentto fill gaps, and facilitate studies on sample characteristics and generalisability in metaresearch.