Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation
Preprint
Abstract
Medical systematic reviews can be very costly and resource intensive. We explore how Large Language Models (LLMs) can support and be trained to perform literature screening when provided with a detailed set of selection criteria. Specifically, we instruction tune LLaMA and Guanaco models to perform abstract screening for medical systematic reviews. Our best model, Bio-SIEVE, outperforms both ChatGPT and trained traditional approaches, and generalises better across medical domains. However, there remains the challenge of adapting the model to safety-first scenarios. We also explore the impact of multi-task training with Bio-SIEVE-Multi, including tasks such as PICO extraction and exclusion reasoning, but find that it is unable to match single-task Bio-SIEVE's performance. We see Bio-SIEVE as an important step towards specialising LLMs for the biomedical systematic review process and explore its future developmental opportunities. We release our models, code and a list of DOIs to reconstruct our dataset for reproducibility.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- A novel application of machine learning and zero-shot classification methods for automated abstract screening in systematic reviewsDecision Analytics Journal, 2023
- Natural language processing was effective in assisting rapid title and abstract screening when updating systematic reviewsJournal of Clinical Epidemiology, 2021
- Rethink Training of BERT Rerankers in Multi-stage Retrieval Pipeline2021
- The significant cost of systematic reviews and meta-analyses: A call for greater involvement of machine learning to assess the promise of clinical trialsContemporary Clinical Trials Communications, 2019
- BioBERT: a pre-trained biomedical language representation model for biomedical text miningBioinformatics, 2019
- Prioritising references for systematic reviews with RobotAnalyst: A user studyResearch Synthesis Methods, 2018
- Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approachJournal of the American Medical Informatics Association, 2017
- Can abstract screening workload be reduced using text mining? User experiences of the tool RayyanResearch Synthesis Methods, 2017
- Reducing systematic review workload through certainty-based screeningJournal of Biomedical Informatics, 2014
- Deploying an interactive machine learning system in an evidence-based practice center2012