Efficiency and Workload Reduction of Semi-automated Citation Screening Software for Creating Clinical Practice Guidelines: A Prospective Observational Study
Open Access
- 5 August 2024
- journal article
- research article
- Published by Japan Epidemiological Association in Journal of Epidemiology
- Vol. 34 (8) , 380-386
Abstract
Background:We evaluated the applicability of automated citation screening in developing clinical practice guidelines.Methods:We prospectively compared the efficiency of citation screening between the conventional (Rayyan) and semi-automated (ASReview software) methods. We searched the literature for five clinical questions (CQs) in the development of the Japanese Clinical Practice Guidelines for the Management of Sepsis and Septic Shock. Objective measurements of the time required to complete citation screening were recorded. Following the first screening round, in the primary analysis, the sensitivity, specificity, positive predictive value, and overall screening time were calculated for both procedures using the semi-automated tool as index and the results of the conventional method as standard reference. In the secondary analysis, the same parameters were compared between the two procedures using the final list of included studies after the second screening session as standard reference.Results:Among the five CQs after the first screening session, the highest and lowest sensitivity, specificity, and positive predictive values were 0.241 and 0.795; 0.991 and 1.000; and 0.482 and 0.929, respectively. In the secondary analysis, the highest sensitivity and specificity in the semi-automated citation screening were 1.000 and 0.997, respectively. The overall screening time per 100 studies was significantly shorter with semi-automated than with conventional citation screening.Conclusions:The potential advantages of the semi-automated method (shorter screening time and higher discriminatory rate for the final list of studies) warrant further validation.Trial registration: This study was submitted to the University Hospital Medical Information Network Clinical Trial Registry (UMIN-CTR [UMIN000049366]).Keywords
Funding Information
- Japan Society for the Promotion of Science (JP23K16253)
This publication has 21 references indexed in Scilit:
- Cochrane Handbook for Systematic Reviews of Interventions2019
- Toward systematic review automation: a practical guide to using machine learning tools in research synthesisSystematic Reviews, 2019
- A question of trust: can we build an evidence base to gain trust in systematic review automation technologies?Systematic Reviews, 2019
- Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning toolSystematic Reviews, 2018
- Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registryBMJ Open, 2017
- RobotReviewer: evaluation of a system for automatically assessing bias in clinical trialsJournal of the American Medical Informatics Association, 2015
- Using text mining for study identification in systematic reviews: a systematic review of current approachesSystematic Reviews, 2015
- QUADAS-2: A Revised Tool for the Quality Assessment of Diagnostic Accuracy StudiesAnnals of Internal Medicine, 2011
- Precision of healthcare systematic review searches in a cross‐sectional sampleResearch Synthesis Methods, 2011
- Semi-automated screening of biomedical citations for systematic reviewsBMC Bioinformatics, 2010