Deep Neural Network for Reducing the Screening Workload in Systematic Reviews for Clinical Guidelines: Algorithm Validation Study (Preprint)
Preprint
- 14 July 2020
- preprint
- Published by JMIR Publications Inc. in JMIR Preprints
Abstract
BACKGROUND Performing systematic reviews is a time-consuming and resource-intensive process. OBJECTIVE We investigated whether a machine learning system could perform systematic reviews more efficiently. METHODS All systematic reviews and meta-analyses of interventional randomized controlled trials cited in recent clinical guidelines from the American Diabetes Association, American College of Cardiology, American Heart Association (2 guidelines), and American Stroke Association were assessed. After reproducing the primary screening data set according to the published search strategy of each, we extracted correct articles (those actually reviewed) and incorrect articles (those not reviewed) from the data set. These 2 sets of articles were used to train a neural network–based artificial intelligence engine (Concept Encoder, Fronteo Inc). The primary endpoint was work saved over sampling at 95% recall (WSS@95%). RESULTS Among 145 candidate reviews of randomized controlled trials, 8 reviews fulfilled the inclusion criteria. For these 8 reviews, the machine learning system significantly reduced the literature screening workload by at least 6-fold versus that of manual screening based on WSS@95%. When machine learning was initiated using 2 correct articles that were randomly selected by a researcher, a 10-fold reduction in workload was achieved versus that of manual screening based on the WSS@95% value, with high sensitivity for eligible studies. The area under the receiver operating characteristic curve increased dramatically every time the algorithm learned a correct article. CONCLUSIONS Concept Encoder achieved a 10-fold reduction of the screening workload for systematic review after learning from 2 randomly selected studies on the target topic. However, few meta-analyses of randomized controlled trials were included. Concept Encoder could facilitate the acquisition of evidence for clinical guidelines.Keywords
All Related Versions
- Published version: Journal of Medical Internet Research, 22 (12), e22422.
This publication has 28 references indexed in Scilit:
- Segmental immune disorders related to herpes zosterInternational Journal of Dermatology, 2013
- Blood pressure targets for hypertension in people with diabetes mellitusCochrane Database of Systematic Reviews, 2013
- New Oral Anticoagulants and the Risk of Intracranial HemorrhageJAMA Neurology, 2013
- Intensive and Standard Blood Pressure Targets in Patients With Type 2 Diabetes MellitusArchives of internal medicine (1960), 2012
- Toward modernizing the systematic review pipeline in genetics: efficient updating via data miningGenetics in Medicine, 2012
- Performance of support-vector-machine-based classification on 15 systematic review topics evaluated with the WSS@95 measure: Table 1Journal of the American Medical Informatics Association, 2011
- Seventy-Five Trials and Eleven Systematic Reviews a Day: How Will We Ever Keep Up?PLOS Medicine, 2010
- Time to Update and Quantitative Changes in the Results of Cochrane Pregnancy and Childbirth ReviewsPLOS ONE, 2010
- Semi-automated screening of biomedical citations for systematic reviewsBMC Bioinformatics, 2010
- Preparing a Cochrane Review2008