Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach
Open Access
- 25 May 2017
- journal article
- research article
- Published by Oxford University Press (OUP) in Journal of the American Medical Informatics Association
- Vol. 24 (6) , 1165-1168
Abstract
Objectives: Identifying all published reports of randomized controlled trials (RCTs) is an important aim, but it requires extensive manual effort to separate RCTs from non-RCTs, even using current machine learning (ML) approaches. We aimed to make this process more efficient via a hybrid approach using both crowdsourcing and ML.Methods: We trained a classifier to discriminate between citations that describe RCTs and those that do not. We then adopted a simple strategy of automatically excluding citations deemed very unlikely to be RCTs by the classifier and deferring to crowdworkers otherwise.Results: Combining ML and crowdsourcing provides a highly sensitive RCT identification strategy (our estimates suggest 95%–99% recall) with substantially less effort (we observed a reduction of around 60%–80%) than relying on manual screening alone.Conclusions: Hybrid crowd-ML strategies warrant further exploration for biomedical curation/annotation tasks.Keywords
Funding Information
- National Institutes of Health
This publication has 10 references indexed in Scilit:
- Combining Crowd and Expert Labels Using Decision Theoretic Active LearningProceedings of the AAAI Conference on Human Computation and Crowdsourcing, 2015
- Automated confidence ranked classification of randomized controlled trial articles: an aid to evidence-based medicineJournal of the American Medical Informatics Association, 2015
- Who Should Label What? Instance Allocation in Multiple Expert Active Learning2011
- Understanding why evidence from randomised clinical trials may not be retrieved from Medline: comparison of indexed and non-indexed recordsBMJ, 2011
- Seventy-Five Trials and Eleven Systematic Reviews a Day: How Will We Ever Keep Up?PLOS Medicine, 2010
- Semi-automated screening of biomedical citations for systematic reviewsBMC Bioinformatics, 2010
- Retrieving randomized controlled trials from medline: a comparison of 38 published search filtersHealth Information and Libraries Journal, 2009
- Reducing Workload in Systematic Review Preparation Using Automated Citation ClassificationJournal of the American Medical Informatics Association, 2006
- Text categorization with Support Vector Machines: Learning with many relevant features1998
- The Cochrane Collaboration: Preparing, Maintaining, and Disseminating Systematic Reviews of the Effects of Health CareAnnals of the New York Academy of Sciences, 1993