Determining relative importance of variables in developing and validating predictive models
Open Access
- 14 September 2009
- journal article
- research article
- Published by Springer Nature in BMC Medical Research Methodology
- Vol. 9 (1) , 64
Abstract
Multiple regression models are used in a wide range of scientific disciplines and automated model selection procedures are frequently used to identify independent predictors. However, determination of relative importance of potential predictors and validating the fitted models for their stability, predictive accuracy and generalizability are often overlooked or not done thoroughly. Using a case study aimed at predicting children with acute lymphoblastic leukemia (ALL) who are at low risk of Tumor Lysis Syndrome (TLS), we propose and compare two strategies, bootstrapping and random split of data, for ordering potential predictors according to their relative importance with respect to model stability and generalizability. We also propose an approach based on relative increase in percentage of explained variation and area under the Receiver Operating Characteristic (ROC) curve for developing models where variables from our ordered list enter the model according to their importance. An additional data set aimed at identifying predictors of prostate cancer penetration is also used for illustrative purposes. Age is chosen to be the most important predictor of TLS. It is selected 100% of the time using the bootstrapping approach. Using the random split method, it is selected 99% of the time in the training data and is significant (at 5% level) 98% of the time in the validation data set. This indicates that age is a stable predictor of TLS with good generalizability. The second most important variable is white blood cell count (WBC). Our methods also identified an important predictor of TLS that was otherwise omitted if relying on any of the automated model selection procedures alone. A group at low risk of TLS consists of children younger than 10 years of age, without T-cell immunophenotype, whose baseline WBC is < 20 × 109/L and palpable spleen is < 2 cm. For the prostate cancer data set, the Gleason score and digital rectal exam are identified to be the most important indicators of whether tumor has penetrated the prostate capsule. Our model selection procedures based on bootstrap re-sampling and repeated random split techniques can be used to assess the strength of evidence that a variable is truly an independent and reproducible predictor. Our methods, therefore, can be used for developing stable and reproducible models with good performances. Moreover, our methods can serve as a good tool for validating a predictive model. Previous biological and clinical studies support the findings based on our selection and validation strategies. However, extensive simulations may be required to assess the performance of our methods under different scenarios as well as check their sensitivity to a random fluctuation in the data.Keywords
This publication has 36 references indexed in Scilit:
- A Prediction Model for Lung Cancer Diagnosis that Integrates Genomic and Clinical FeaturesCancer Prevention Research, 2008
- Evaluation of Logistic Regression Reporting in Current Obstetrics and Gynecology LiteratureObstetrics & Gynecology, 2008
- Features at presentation predict children with acute lymphoblastic leukemia at low risk for tumor lysis syndromeCancer, 2007
- How Well Does the Gleason Score Predict Prostate Cancer Death? A 20-Year Followup of a Population Based Cohort in SwedenJournal of Urology, 2006
- A predictive model for the detection of tumor lysis syndrome during AML induction therapyLeukemia & Lymphoma, 2006
- Can the strengths of AIC and BIC be shared? A conflict between model indentification and regression estimationBiometrika, 2005
- Integrated modeling of clinical and gene expression information for personalized prediction of disease outcomesProceedings of the National Academy of Sciences, 2004
- Towards integrated clinico-genomic models for personalized medicine: combining gene expression signatures and clinical factors in breast cancer outcomes predictionHuman Molecular Genetics, 2003
- Predictive accuracy and explained variationStatistics in Medicine, 2003
- Methods of variable selection in regression modelingCommunications in Statistics - Simulation and Computation, 1998