Machine learning for identifying relevant publications in updates of systematic reviews of diagnostic test studies
Open Access
- 15 March 2021
- journal article
- research article
- Published by Wiley in Research Synthesis Methods
- Vol. 12 (4) , 506-515
Abstract
Updating systematic reviews is often a time‐consuming process that involves a lot of human effort and is therefore not conducted as often as it should be. The aim of our research project was to explore the potential of machine learning methods to reduce human workload. Furthermore, we evaluated the performance of deep learning methods in comparison to more established machine learning methods. We used three available reviews of diagnostic test studies as the data set. In order to identify relevant publications, we used typical text pre‐processing methods. The reference standard for the evaluation was the human‐consensus based on binary classification (inclusion, exclusion). For the evaluation of the models, various scenarios were generated using a grid of combinations of data preprocessing steps. Moreover, we evaluated each machine learning approach with an approach‐specific predefined grid of tuning parameters using the Brier score metric. The best performance was obtained with an ensemble method for two of the reviews, and by a deep learning approach for the other review. Yet, the final performance of approaches strongly depends on data preparation. Overall, machine learning methods provided reasonable classification. It seems possible to reduce human workload in updating systematic reviews by using machine learning methods. Yet, as the influence of data preprocessing on the final performance seems to be at least as important as choosing the specific machine learning approach, users should not blindly expect a good performance by solely using approaches from a popular class, such as deep learning.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- The Brier score does not evaluate the clinical utility of diagnostic tests or prediction modelsDiagnostic and Prognostic Research, 2017
- Living systematic reviews: 2. Combining human and machine effortJournal of Clinical Epidemiology, 2017
- The reliability of physical examination tests for the clinical assessment of scapular dyskinesis in subjects with shoulder complaints: A systematic reviewPhysical Therapy in Sport, 2017
- Reliability of specific physical examination tests for the diagnosis of shoulder pathologies: a systematic review and meta-analysisBritish Journal of Sports Medicine, 2016
- Physical tests for diagnosing anterior cruciate ligament ruptureCochrane Database of Systematic Reviews, 2015
- Deep learningNature, 2015
- Using text mining for study identification in systematic reviews: a systematic review of current approachesSystematic Reviews, 2015
- Reporting quality of diagnostic accuracy studies: a systematic review and meta-analysis of investigations on adherence to STARDBMJ Evidence-Based Medicine, 2013
- Physical Examination Tests for the Diagnosis of Posterior Cruciate Ligament Rupture: A Systematic ReviewJournal of Orthopaedic & Sports Physical Therapy, 2013
- The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and ElaborationPLOS Medicine, 2009