Machine learning reduced workload for the Cochrane COVID-19 Study Register: development and evaluation of the Cochrane COVID-19 Study Classifier
Authors:
Open Access
- 22 January 2022
- journal article
- research article
- Published by Springer Nature in Systematic Reviews
- Vol. 11 (1) , 1-8
Abstract
Background: This study developed, calibrated and evaluated a machine learning (ML) classifier designed to reduce study identification workload in maintaining the Cochrane COVID-19 Study Register (CCSR), a continuously updated register of COVID-19 research studies. Methods: A ML classifier for retrieving COVID-19 research studies (the ‘Cochrane COVID-19 Study Classifier’) was developed using a data set of title-abstract records ‘included’ in, or ‘excluded’ from, the CCSR up to 18th October 2020, manually labelled by information and data curation specialists or the Cochrane Crowd. The classifier was then calibrated using a second data set of similar records ‘included’ in, or ‘excluded’ from, the CCSR between October 19 and December 2, 2020, aiming for 99% recall. Finally, the calibrated classifier was evaluated using a third data set of similar records ‘included’ in, or ‘excluded’ from, the CCSR between the 4th and 19th of January 2021. Results: The Cochrane COVID-19 Study Classifier was trained using 59,513 records (20,878 of which were ‘included’ in the CCSR). A classification threshold was set using 16,123 calibration records (6005 of which were ‘included’ in the CCSR) and the classifier had a precision of 0.52 in this data set at the target threshold recall >0.99. The final, calibrated COVID-19 classifier correctly retrieved 2285 (98.9%) of 2310 eligible records but missed 25 (1%), with a precision of 0.638 and a net screening workload reduction of 24.1% (1113 records correctly excluded). Conclusions: The Cochrane COVID-19 Study Classifier reduces manual screening workload for identifying COVID-19 research studies, with a very low and acceptable risk of missing eligible studies. It is now deployed in the live study identification workflow for the Cochrane COVID-19 Study Register.Keywords
All Related Versions
Funding Information
- Cochrane (NA)
This publication has 9 references indexed in Scilit:
- Evaluation of the comprehensiveness, accuracy and currency of the Cochrane COVID‐19 Study Register for supporting rapid evidence synthesis productionResearch Synthesis Methods, 2021
- An evaluation of Cochrane Crowd found that crowdsourcing produced accurate results in identifying randomized trialsJournal of Clinical Epidemiology, 2021
- Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane ReviewsJournal of Clinical Epidemiology, 2021
- COVID-19-related medical research: a meta-research and critical appraisalBMC Medical Research Methodology, 2021
- How a torrent of COVID science changed research publishing — in seven chartsNature, 2020
- Collaborating in response to COVID-19: editorial and methods initiatives across Cochrane2020
- Cochrane Centralised Search Service showed high sensitivity identifying randomized controlled trials: A retrospective analysisJournal of Clinical Epidemiology, 2020
- The runaway science: a bibliometric analysis of the COVID-19 scientific literature.2020
- Using text mining for study identification in systematic reviews: a systematic review of current approachesSystematic Reviews, 2015