Multiple Versus Single Set Validation of Multivariate Models to Avoid Mistakes
- 25 October 2017
- journal article
- review article
- Published by Taylor & Francis in Critical Reviews in Analytical Chemistry
- Vol. 48 (1) , 33-46
Abstract
Validation of multivariate models is of current importance for a wide range of chemical applications. Although important, it is neglected. The common practice is to use a single external validation set for evaluation. This approach is deficient and may mislead investigators with results that are specific to the single validation set of data. In addition, no statistics are available regarding the precision of a derived figure of merit (FOM). A statistical approach using bootstrapped Latin partitions is advocated. This validation method makes an efficient use of the data because each object is used once for validation. It was reviewed a decade earlier but primarily for the optimization of chemometric models this review presents the reasons it should be used for generalized statistical validation. Average FOMs with confidence intervals are reported and powerful, matched-sample statistics may be applied for comparing models and methods. Examples demonstrate the problems with single validation sets.Keywords
This publication has 55 references indexed in Scilit:
- Sparse Partial Least Squares Regression for Simultaneous Dimension Reduction and Variable SelectionJournal of the Royal Statistical Society Series B: Statistical Methodology, 2010
- Bootstrap classification and point-based feature selection from age-staged mouse cerebellum tissues of matrix assisted laser desorption/ionization mass spectra using a fuzzy rule-building expert systemAnalytica Chimica Acta, 2007
- Statistical validation of classification and calibration models using bootstrapped Latin partitionsTrAC Trends in Analytical Chemistry, 2006
- Representative subset selectionAnalytica Chimica Acta, 2002
- Screening GC-MS data for carbamate pesticides with temperature-constrained–cascade correlation neural networksAnalytica Chimica Acta, 2000
- OptiSim: An Extended Dissimilarity Selection Method for Finding Diverse Representative SubsetsJournal of Chemical Information and Computer Sciences, 1997
- Bootstrap Model SelectionJournal of the American Statistical Association, 1996
- Linear Model Selection by Cross-validationJournal of the American Statistical Association, 1993
- Validation of Regression Models: Methods and ExamplesTechnometrics, 1977
- Computer Aided Design of ExperimentsTechnometrics, 1969