Estimation of predictive performance for test data in applicability domains using y‐randomization
- 29 July 2019
- journal article
- research article
- Published by Wiley in Journal of Chemometrics
- Vol. 33 (9) , e3171
Abstract
A new measure of predictive performance for objective variables in regression analysis is proposed, enabling the y‐errors of new samples or test samples to be estimated in the applicability domains (ADs) of regression models. The proposed measure, based on y‐randomization, considers chance correlations and is calculated using only training data. This chance correlation‐excluded mean absolute error (MAECCE) can estimate the y‐errors of new samples considering the influence of chance correlations in the given dataset on the regression models. Experiments using numerical simulation, quantitative structure‐activity relationship, and quantitative structure‐property relationship datasets confirm that MAECCE can estimate the distribution of y‐errors of new samples in ADs for various training datasets, descriptor sets, and regression analysis methods, enabling chance correlations to be eliminated from data analysis. Python and MATLAB codes for the proposed algorithm are available at https://github.com/hkaneko1985/maecce.Keywords
This publication has 25 references indexed in Scilit:
- The One‐Class Classification Approach to Data Description and to Models Applicability DomainMolecular Informatics, 2010
- Principles of Proper Validation: use and abuse of re‐sampling for validationJournal of Chemometrics, 2010
- Predicting the Predictability: A Unified Approach to the Applicability Domain Problem of QSAR ModelsJournal of Chemical Information and Modeling, 2009
- Repeated double cross validationJournal of Chemometrics, 2009
- Critical Assessment of QSAR Models of Environmental Toxicity against Tetrahymena pyriformis: Focusing on Applicability Domain and Overfitting by Variable SelectionJournal of Chemical Information and Modeling, 2008
- Random Forest Models To Predict Aqueous SolubilityJournal of Chemical Information and Modeling, 2006
- Pattern Recognition and Machine Learning2006
- ADME Evaluation in Drug Discovery. 4. Prediction of Aqueous Solubility Based on Atom Contribution ApproachJournal of Chemical Information and Computer Sciences, 2003
- Assessing Model Fit by Cross-ValidationJournal of Chemical Information and Computer Sciences, 2003
- A Brief Survey of Bandwidth Selection for Density EstimationJournal of the American Statistical Association, 1996