Lori-Ann Linkins

Background: The COVID-19 pandemic demanded evidence synthesis approaches that could keep pace with rapidly evolving science and urgent policy needs. Rapid reviews and living evidence syntheses (LES) emerged as critical methodological responses, yet limited guidance exists for teams undertaking such work, particularly regarding the operational challenges of sustained production, multi-team coordination, and the transition from rapid reviews to living formats under emergency conditions. Objectives: This paper aims to document and appraise methodological decisions made across each component of the synthesis process and their rationale; identify the contextual factors, coordination mechanisms, and infrastructure that enabled or constrained synthesis implementation under emergency conditions; and derive practical recommendations for evidence synthesis teams, funders and commissioners, and methodologists. Methods: We present a structured reflective process evaluation of our experience producing three rapid LESs on COVID-19 vaccine effectiveness as part of COVID-END in Canada, addressing vaccine effectiveness in adults (short-term and long-term) and children/adolescents, collectively producing 79 versions over approximately two years. Drawing on a Context–Mechanism–Outcome (CMO) lens, we examine eight core methodological processes, each time highlighting: the methodological decision and its rationale; the challenges encountered; and the lessons learned. Findings: Key insights include: decision-maker engagement played a central role in methodological decisions, directly determining scope boundaries, update frequency, and presentation formats; the multi-team coordination structure was essential to quality under streamlined methods, with cross-team verification partially offsetting the validity risks of single-reviewer screening; choosing between narrative synthesis (compared to meta-analytic approaches) in two of three products reflected capacity constraints during emergency production, a distinction with implications for how such adaptations are evaluated against existing guidance; automation produced considerable efficiency gains but required added development capacity and resources, demonstrating that teams should plan and budget for automation from the outset rather than absorbing its demands reactively during production; and coverage was acceptable for decision timelines. External cross-checks identified five studies not captured by our searches across the entire project, of which only two were ultimately eligible for synthesis. Emergency production meant that key process records, including change logs, per-cycle search records, and pilot screening agreement data, were not systematically captured. Future teams should plan to assemble a minimal transparency package from the outset. A minimum viable LES framework is proposed comprising minimum infrastructure, minimum methods, and optional enhancers. Conclusions: The successful implementation of rapid LES requires pre-existing or rapidly mobilisable infrastructure, sustained and explicitly resourced decision-maker engagement, transparent documentation of methodological adaptations across update cycles, effective coordination in multi-team contexts, and realistic sustainability planning. These lessons, derived from a specific emergency context, offer transferable guidance for evidence synthesis teams preparing for future health emergencies and for standards bodies considering how existing guidance accommodates emergency and living synthesis conditions.

The COVID-19 pandemic demanded evidence synthesis approaches that could keep pace with rapidly evolving science and urgent policy needs. Rapid reviews and living evidence syntheses (LES) emerged as critical methodological responses, yet limited guidance exists for teams undertaking such work, particularly regarding the operational challenges of sustained production, multi-team coordination, and the transition from rapid reviews to living formats under emergency conditions. This paper aims to document and appraise methodological decisions made across each component of the synthesis process and their rationale; identify the contextual factors, coordination mechanisms, and infrastructure that enabled or constrained synthesis implementation under emergency conditions; and derive practical recommendations for evidence synthesis teams, funders and commissioners, and methodologists. We present a structured reflective process evaluation of our experience producing three rapid LESs on COVID-19 vaccine effectiveness as part of COVID-END in Canada, addressing vaccine effectiveness in adults (short-term and long-term) and children/adolescents, collectively producing 79 versions over approximately two years. Drawing on a Context–Mechanism–Outcome (CMO) lens, we examine eight core methodological processes, each time highlighting: the methodological decision and its rationale; the challenges encountered; and the lessons learned. Key insights include: decision-maker engagement played a central role in methodological decisions, directly determining scope boundaries, update frequency, and presentation formats; the multi-team coordination structure was essential to quality under streamlined methods, with cross-team verification partially offsetting the validity risks of single-reviewer screening; choosing between narrative synthesis (compared to meta-analytic approaches) in two of three products reflected capacity constraints during emergency production, a distinction with implications for how such adaptations are evaluated against existing guidance; automation produced considerable efficiency gains but required added development capacity and resources, demonstrating that teams should plan and budget for automation from the outset rather than absorbing its demands reactively during production; and coverage was acceptable for decision timelines. External cross-checks identified five studies not captured by our searches across the entire project, of which only two were ultimately eligible for synthesis. Emergency production meant that key process records, including change logs, per-cycle search records, and pilot screening agreement data, were not systematically captured. Future teams should plan to assemble a minimal transparency package from the outset. A minimum viable LES framework is proposed comprising minimum infrastructure, minimum methods, and optional enhancers. The successful implementation of rapid LES requires pre-existing or rapidly mobilisable infrastructure, sustained and explicitly resourced decision-maker engagement, transparent documentation of methodological adaptations across update cycles, effective coordination in multi-team contexts, and realistic sustainability planning. These lessons, derived from a specific emergency context, offer transferable guidance for evidence synthesis teams preparing for future health emergencies and for standards bodies considering how existing guidance accommodates emergency and living synthesis conditions.

Evidence-based medicine (EBM) emerged from McMaster University in the 1980-1990s, which emphasizes the integration of the best research evidence with clinical expertise and patient values. The Health Information Research Unit (HiRU) was created at McMaster University in 1985 to support EBM. Early on, digital health informatics took the form of teaching clinicians how to search MEDLINE with modems and phone lines. Searching and retrieval of published articles were transformed as electronic platforms provided greater access to clinically relevant studies, systematic reviews, and clinical practice guidelines, with PubMed playing a pivotal role. In the early 2000s, the HiRU introduced Clinical Queries—validated search filters derived from the curated, gold-standard, human-appraised Hedges dataset—to enhance the precision of searches, allowing clinicians to hone their queries based on study design, population, and outcomes. Currently, almost 1 million articles are added to PubMed annually. To filter through this volume of heterogenous publications for clinically important articles, the HiRU team and other researchers have been applying classical machine learning, deep learning, and, increasingly, large language models (LLMs). These approaches are built upon the foundation of gold-standard annotated datasets and humans in the loop for active machine learning. In this viewpoint, we explore the evolution of health informatics in supporting evidence search and retrieval processes over the past 25+ years within the HiRU, including the evolving roles of LLMs and responsible artificial intelligence, as we continue to facilitate the dissemination of knowledge, enabling clinicians to integrate the best available evidence into their clinical practice.

UNSTRUCTURED Evidence-based medicine (EBM) emerged from McMaster University in the 1980-1990s, which emphasizes the integration of the best research evidence with clinical expertise and patient values. The Health Information Research Unit (HiRU) was created at McMaster University in 1985 to support EBM. Early on, digital health informatics took the form of teaching clinicians how to search MEDLINE with modems and phone lines. Searching and retrieval of published articles were transformed as electronic platforms provided greater access to clinically relevant studies, systematic reviews, and clinical practice guidelines, with PubMed playing a pivotal role. In the early 2000s, the HiRU introduced Clinical Queries—validated search filters derived from the curated, gold-standard, human-appraised Hedges dataset—to enhance the precision of searches, allowing clinicians to hone their queries based on study design, population, and outcomes. Currently, almost 1 million articles are added to PubMed annually. To filter through this volume of heterogenous publications for clinically important articles, the HiRU team and other researchers have been applying classical machine learning, deep learning, and, increasingly, large language models (LLMs). These approaches are built upon the foundation of gold-standard annotated datasets and humans in the loop for active machine learning. In this viewpoint, we explore the evolution of health informatics in supporting evidence search and retrieval processes over the past 25+ years within the HiRU, including the evolving roles of LLMs and responsible artificial intelligence, as we continue to facilitate the dissemination of knowledge, enabling clinicians to integrate the best available evidence into their clinical practice.

BACKGROUND Given suboptimal performance of Boolean searching to identify methodologically sound and clinically relevant studies in large bibliographic databases such as MEDLINE, exploring the performance of machine learning (ML) tools is warranted. OBJECTIVE Using a large internationally recognized dataset of articles tagged for methodological rigor, we trained and tested binary classification models to predict the probability of clinical research articles being of high methodologic quality to support a literature surveillance program. METHODS Over 12,000 models were trained on a dataset of 97,805 articles indexed in PubMed from 2012-2018 which were manually appraised for rigor by highly trained research associates with expertise in research methods and critical appraisal. As the dataset is unbalanced, with more articles that do not meet criteria for rigor, we used the unbalanced dataset and over- and under-sampled datasets. Models that maintained sensitivity for high rigor at 99% and maximized specificity were selected and tested in a retrospective set of 30,424 articles from 2020 and validated prospectively in a blinded study of 5253 articles. RESULTS The final selected algorithm, combining a model trained in each dataset, maintained high sensitivity and achieved 57% specificity in the retrospective validation test and 53% in the prospective study. The number of articles needed to read to find one that met appraisal criteria was 3.68 (3.52 to 3.85) in the prospective study. CONCLUSIONS ML models improved the efficiency and precision of detecting high quality clinical research publications for literature surveillance and subsequent dissemination to clinicians and other evidence users.

Background: A barrier to practicing evidence-based medicine is the rapidly increasing body of biomedical literature. Use of method terms to limit the search can help reduce the burden of screening articles for clinical relevance; however, such terms are limited by their partial dependence on indexing terms and usually produce low precision, especially when high sensitivity is required. Machine learning has been applied to the identification of high-quality literature with the potential to achieve high precision without sacrificing sensitivity. The use of artificial intelligence has shown promise to improve the efficiency of identifying sound evidence. Objective: The primary objective of this research is to derive and validate deep learning machine models using iterations of Bidirectional Encoder Representations from Transformers (BERT) to retrieve high-quality, high-relevance evidence for clinical consideration from the biomedical literature. Methods: Using the HuggingFace Transformers library, we will experiment with variations of BERT models, including BERT, BioBERT, BlueBERT, and PubMedBERT, to determine which have the best performance in article identification based on quality criteria. Our experiments will utilize a large data set of over 150,000 PubMed citations from 2012 to 2020 that have been manually labeled based on their methodological rigor for clinical use. We will evaluate and report on the performance of the classifiers in categorizing articles based on their likelihood of meeting quality criteria. We will report fine-tuning hyperparameters for each model, as well as their performance metrics, including recall (sensitivity), specificity, precision, accuracy, F-score, the number of articles that need to be read before finding one that is positive (meets criteria), and classification probability scores. Results: Initial model development is underway, with further development planned for early 2022. Performance testing is expected to star in February 2022. Results will be published in 2022. Conclusions: The experiments will aim to improve the precision of retrieving high-quality articles by applying a machine learning classifier to PubMed searching. International Registered Report Identifier (IRRID): DERR1-10.2196/29398

BACKGROUND A barrier to practicing evidence-based medicine is the rapidly increasing body of biomedical literature. Use of method terms to limit the search can help reduce the burden of screening articles for clinical relevance; however, such terms are limited by their partial dependence on indexing terms and usually produce low precision, especially when high sensitivity is required. Machine learning has been applied to the identification of high-quality literature with the potential to achieve high precision without sacrificing sensitivity. The use of artificial intelligence has shown promise to improve the efficiency of identifying sound evidence. OBJECTIVE The primary objective of this research is to derive and validate deep learning machine models using iterations of Bidirectional Encoder Representations from Transformers (BERT) to retrieve high-quality, high-relevance evidence for clinical consideration from the biomedical literature. METHODS Using the HuggingFace Transformers library, we will experiment with variations of BERT models, including BERT, BioBERT, BlueBERT, and PubMedBERT, to determine which have the best performance in article identification based on quality criteria. Our experiments will utilize a large data set of over 150,000 PubMed citations from 2012 to 2020 that have been manually labeled based on their methodological rigor for clinical use. We will evaluate and report on the performance of the classifiers in categorizing articles based on their likelihood of meeting quality criteria. We will report fine-tuning hyperparameters for each model, as well as their performance metrics, including recall (sensitivity), specificity, precision, accuracy, F-score, the number of articles that need to be read before finding one that is positive (meets criteria), and classification probability scores. RESULTS Initial model development is underway, with further development planned for early 2022. Performance testing is expected to star in February 2022. Results will be published in 2022. CONCLUSIONS The experiments will aim to improve the precision of retrieving high-quality articles by applying a machine learning classifier to PubMed searching. INTERNATIONAL REGISTERED REPORT DERR1-10.2196/29398

OBJECTIVES: Venous thromboembolic events can be successfully prevented with chemical and/or mechanical prophylaxis measures, but evidence-based guidelines in thoracic surgery are limited, particularly regarding extended post-discharge prophylaxis. This study attempts to gather an international consensus on best practices to inform the development of such guidelines. METHODS: A series of 3 surveys was distributed to the ESTS/AATS/ISTH (European Society of Thoracic Surgeons, American Association of Thoracic Surgeons, International Society for Thrombosis and Haemostasis) venous thromboembolic events prophylaxis working group starting January 2017. This iterative Delphi consensus process sought to gather a consensus on (i) risk factors; (ii) preferred agents; (iii) duration; and (iv) perceived barriers to an extended thromboprophylaxis approach. Participant responses were expressed on a 10-point scale, and the results were summarized and circulated to all respondents in subsequent rounds. A coefficient of variance of ≤0.3 was identified pre hoc to identify agreement. RESULTS: A total of 21 Working Group members completed the surveys, composed of 19% non-surgeon thrombosis experts, and 48% from North America. Respondents largely saw agreement regarding risk factors that indicate a need for extended thromboprophylaxis. The group agreed that low-molecular-weight heparin is a suitable agent for use post-discharge, but there was a wide variety in response regarding agents, duration and barriers to extended prophylaxis, where no consensus was observed across the three rounds. CONCLUSIONS: There is strong agreement around indications for extended venous thromboembolic events thromboprophylaxis after thoracic surgery, but there is little consensus regarding the agents and duration to be employed. Further research is required to better inform guideline development.