Bunga Citra Pratiwi

In a psychometric analysis of a new psychological test, we often assess the predictive validity of a new target test over and above a baseline test, known as the incremental predictive validity. Usually, the incremental predictive validity is evaluated using within-sample statistics. Recently, it was argued to use out-of-sample assessment to prevent overfitting and non-replicable findings. In this paper, we elaborate on how to assess incremental predictive validity out-of-sample. In such an approach, we estimate prediction rules in one sample, and evaluate incremental predictive validity in another sample. Using a simulation study, we investigate whether an out-of-sample assessment results in different findings than a within-sample evaluation, taking into account the reliability of the baseline and a target test, and other factors (i.e., sample size). Results show that there is a difference between the in-sample and out-of-sample assessment, especially in small samples. However, the reliability of the two tests has no influence on this difference. In addition, we explore the effects of ridge estimation, ordinary least squares, and SIMEX, three different methods for estimating a prediction rule, on incremental predictive validity. The results show that using SIMEX leads to a bad assessment of incremental predictive validity. Ordinary least squares and ridge estimation result in almost the same incremental predictive validity estimates with a little advantage for ridge regression. In an empirical application, we show how to assess incremental predictive validity in practice and we compare that to the usual assessment.

Predictive modeling is becoming more popular in psychological science. Machine learning techniques have been used to develop prediction rules based on items of psychological tests. However, this approach does not take into account that these items are noisy indicators of the constructs they intend to measure. Structural equation modeling does take this into account. Several authors have concluded that it is impossible to make out-of-sample predictions based on a reflective structural equation model. We show that it is possible to make such predictions and we develop R-code to do so. With two empirical examples, we show that SEM-based prediction can outperform prediction based on linear regression models. With three simulation studies, we further investigate the SEM-based prediction rule and its robustness in comparison with predictions using regularized linear regression. We conclude that the new SEM-based prediction rule is robust against violation of the normality assumption but sensitive to model misspecification.

When validating a new test, its incremental predictive validity (IV) is often evaluatedfrom a classical approach using in-sample estimates. In this study, to evaluate IV, weadopt a statistical learning (SL) approach that uses out-of-sample estimates. Followingthis approach, we identify two important aspects to consider: 1) evaluate the predictionrules based on the out-of-sample prediction error, and 2) obtain the prediction rulesusing different estimation methods (e.g., OLS, SIMEX, and ridge). We performed asimulation study to investigate the effect of using in- and out-of-sample measures on IV,and to assess the influence of the reliabilities of tests on these two estimates of IV.Results showed that the two approaches differed, such that in-sample IV is on averagelarger than out-of-sample IV, but that the difference decreases as the size of thecalibration sample increases. In- and out-of-sample IV were both affected by thereliability of the added test, that is, as reliability increases IV also increases. Inaddition, if collinearity exists between two tests, the reliability of the first test has anegative relationship with the IV of the second test. Worthy of note is that thedirection of these effects did not differ between the two approaches (in/out-of-sample).It was further shown that using ridge regression enhances IV defined as out-of-samplewhen the size of the calibration sample is small. We also showed how to apply the newprinciples in practice using empirical data from a student selection procedure. Weconcluded that in addition to establishing strong reliability for the new test, validationshould be based on out-of-sample prediction accuracy and that users should consider ashrinkage method to estimate optimal rules for prediction for new persons.