Machine learning for identifying Randomized Controlled Trials: An evaluation and practitioner's guide
Top Cited Papers
Open Access
- 4 January 2018
- journal article
- research article
- Published by Wiley in Research Synthesis Methods
- Vol. 9 (4) , 602-614
Abstract
Machine learning (ML) algorithms have proven highly accurate for identifying Randomized Controlled Trials (RCTs) but are not used much in practice, in part because the best way to make use of the technology in a typical workflow is unclear. In this work, we evaluate ML models for RCT classification (support vector machines, convolutional neural networks, and ensemble approaches). We trained and optimized support vector machine and convolutional neural network models on the titles and abstracts of theCochrane CrowdRCT set. We evaluated the models on an external dataset (Clinical Hedges), allowing direct comparison with traditional database search filters. We estimated area under receiver operating characteristics (AUROC) using the Clinical Hedges dataset.We demonstrate that ML approaches better discriminate between RCTs and non‐RCTs than widely used traditional database search filters at all sensitivity levels; our best‐performing model also achieved the best results to date for ML in this task (AUROC 0.987, 95% CI, 0.984‐0.989). We provide practical guidance on the role of ML in (1) systematic reviews (high‐sensitivity strategies) and (2) rapid reviews and clinical question answering (high‐precision strategies) together with recommended probability cutoffs for each use case. Finally, we provide open‐source software to enable these approaches to be used in practice.Keywords
Funding Information
- Medical Research Council (MR/N015185/1)
- U.S. National Library of Medicine (5UH2CA203711‐02,, R01‐LM012086‐01A1)
- Agency for Healthcare Research and Quality (R03‐HS025024)
This publication has 22 references indexed in Scilit:
- Measuring the Performance of the Cochrane Library2012
- pROC: an open-source package for R and S+ to analyze and compare ROC curvesBMC Bioinformatics, 2011
- Retrieving randomized controlled trials from medline: a comparison of 38 published search filtersHealth Information and Libraries Journal, 2009
- Tips for learners of evidence-based medicine: 1. Relative risk reduction, absolute risk reduction and number needed to treatCanadian Medical Association Journal, 2004
- Ensemble Methods in Machine Learning2000
- Text categorization with Support Vector Machines: Learning with many relevant features1998
- Improving the Quality of Reporting of Randomized Controlled TrialsJAMA, 1996
- Improving the quality of reporting of randomized controlled trials. The CONSORT statementJAMA, 1996
- Bagging predictorsMachine Learning, 1996
- Term-weighting approaches in automatic text retrievalInformation Processing & Management, 1988