Semi-automating abstract screening with a natural language model pretrained on biomedical literature
Open Access
- 23 September 2023
- journal article
- letter
- Published by Springer Nature in Systematic Reviews
- Vol. 12 (1) , 1-3
Abstract
We demonstrate the performance and workload impact of incorporating a natural language model, pretrained on citations of biomedical literature, on a workflow of abstract screening for studies on prognostic factors in end-stage lung disease. The model was optimized on one-third of the abstracts, and model performance on the remaining abstracts was reported. Performance of the model, in terms of sensitivity, precision, F1 and inter-rater agreement, was moderate in comparison with other published models. However, incorporating it into the screening workflow, with the second reviewer screening only abstracts with conflicting decisions, translated into a 65% reduction in the number of abstracts screened by the second reviewer. Subsequent work will look at incorporating the pre-trained BERT model into screening workflows for other studies prospectively, as well as improving model performance.Keywords
Funding Information
- National Medical Research Council, Singapore (HSRGEoL18may-0003)
This publication has 10 references indexed in Scilit:
- Prognostic Factors of Mortality in Nonchronic Obstructive Pulmonary Disease Chronic Lung Disease: A Scoping ReviewJournal of Palliative Medicine, 2024
- Automated medical literature screening using artificial intelligence: a systematic review and meta-analysisJournal of the American Medical Informatics Association, 2022
- Using artificial intelligence methods for systematic review in health sciences: A systematic reviewResearch Synthesis Methods, 2022
- srBERT: automatic article classification model for systematic review using BERTSystematic Reviews, 2021
- Natural language processing was effective in assisting rapid title and abstract screening when updating systematic reviewsJournal of Clinical Epidemiology, 2021
- Aligning text mining and machine learning algorithms with best practices for study selection in systematic literature reviewsSystematic Reviews, 2020
- Decoding semi-automated title-abstract screening: findings from a convenience sample of reviewsSystematic Reviews, 2020
- Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three toolsSystematic Reviews, 2019
- Performance and Usability of Machine Learning for Screening in Systematic Reviews: A Comparative Evaluation of Three Tools2019
- Toward systematic review automation: a practical guide to using machine learning tools in research synthesisSystematic Reviews, 2019