Using Background Knowledge from Preceding Studies for Building a Random Forest Prediction Model: A Plasmode Simulation Study
Open Access
Abstract
There is an increasing interest in machine learning (ML) algorithms for predicting patient outcomes, as these methods are designed to automatically discover complex data patterns. For example, the random forest (RF) algorithm is designed to identify relevant predictor variables out of a large set of candidates. In addition, researchers may also use external information for variable selection to improve model interpretability and variable selection accuracy, thereby prediction quality. However, it is unclear to which extent, if at all, RF and ML methods may benefit from external information. In this paper, we examine the usefulness of external information from prior variable selection studies that used traditional statistical modeling approaches such as the Lasso, or suboptimal methods such as univariate selection. We conducted a plasmode simulation study based on subsampling a data set from a pharmacoepidemiologic study with nearly 200,000 individuals, two binary outcomes and 1152 candidate predictor (mainly sparse binary) variables. When the scope of candidate predictors was reduced based on external knowledge RF models achieved better calibration, that is, better agreement of predictions and observed outcome rates. However, prediction quality measured by cross-entropy, AUROC or the Brier score did not improve. We recommend appraising the methodological quality of studies that serve as an external information source for future prediction model development.Keywords
Funding Information
- Austrian Science Fund (I4739-B)
This publication has 33 references indexed in Scilit:
- Potential Savings in Prescription Drug Costs for Hypertension, Hyperlipidemia, and Diabetes Mellitus by Equivalent Drug Substitution in Austria: A Nationwide Cohort StudyApplied Health Economics and Health Policy, 2014
- Prognosis research strategy (PROGRESS) 1: A framework for researching clinical outcomesBMJ, 2013
- Probability MachinesMethods of Information in Medicine, 2012
- Weighted Lasso with Data IntegrationStatistical Applications in Genetics and Molecular Biology, 2011
- To Explain or to Predict?Statistical Science, 2010
- Assessing the Performance of Prediction ModelsEpidemiology, 2010
- Boosting for high-dimensional linear modelsThe Annals of Statistics, 2006
- Regularization and Variable Selection Via the Elastic NetJournal of the Royal Statistical Society Series B: Statistical Methodology, 2005
- Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author)Statistical Science, 2001
- Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysisJournal of Clinical Epidemiology, 1996