Screening PubMed abstracts: is class imbalance always a challenge to machine learning?
Open Access
- 6 December 2019
- journal article
- research article
- Published by Springer Nature in Systematic Reviews
- Vol. 8 (1) , 1-9
Abstract
The growing number of medical literature and textual data in online repositories led to an exponential increase in the workload of researchers involved in citation screening for systematic reviews. This work aims to combine machine learning techniques and data preprocessing for class imbalance to identify the outperforming strategy to screen articles in PubMed for inclusion in systematic reviews. We trained four binary text classifiers (support vector machines, k-nearest neighbor, random forest, and elastic-net regularized generalized linear models) in combination with four techniques for class imbalance: random undersampling and oversampling with 50:50 and 35:65 positive to negative class ratios and none as a benchmark. We used textual data of 14 systematic reviews as case studies. Difference between cross-validated area under the receiver operating characteristic curve (AUC-ROC) for machine learning techniques with and without preprocessing (delta AUC) was estimated within each systematic review, separately for each classifier. Meta-analytic fixed-effect models were used to pool delta AUCs separately by classifier and strategy. Cross-validated AUC-ROC for machine learning techniques (excluding k-nearest neighbor) without preprocessing was prevalently above 90%. Except for k-nearest neighbor, machine learning techniques achieved the best improvement in conjunction with random oversampling 50:50 and random undersampling 35:65. Resampling techniques slightly improved the performance of the investigated machine learning techniques. From a computational perspective, random undersampling 35:65 may be preferred.Keywords
This publication has 21 references indexed in Scilit:
- A machine learning-based framework to identify type 2 diabetes through electronic health recordsInternational Journal of Medical Informatics, 2017
- Text Classification Models for Web Content Filtering and Online Safety2015
- Learning to identify relevant studies for systematic reviews using random forest and external informationMachine Learning, 2015
- Using text mining for study identification in systematic reviews: a systematic review of current approachesSystematic Reviews, 2015
- Reducing systematic review workload through certainty-based screeningJournal of Biomedical Informatics, 2014
- Biologic Therapies in Rheumatoid Arthritis and the Risk of Opportunistic Infections: A Meta-analysisClinical Infectious Diseases, 2014
- Utilization and Perceived Problems of Online Medical Resources and Search Tools Among Different Groups of European PhysiciansJournal of Medical Internet Research, 2013
- Evaluating the effect of unbalanced data in biomedical document classification.2011
- Semi-automated screening of biomedical citations for systematic reviewsBMC Bioinformatics, 2010
- Neighbor-weighted K-nearest neighbor for unbalanced text corpusExpert Systems with Applications, 2005