Prediction of true test scores from observed item scores and ancillary data
- 13 March 2015
- journal article
- Published by Wiley in British Journal of Mathematical and Statistical Psychology
- Vol. 68 (2) , 363-385
Abstract
In many educational tests which involve constructed responses, a traditional test score is obtained by adding together item scores obtained through holistic scoring by trained human raters. For example, this practice was used until 2008 in the case of GRE ® G eneral A nalytical W riting and until 2009 in the case of TOEFL ® i BT W riting. With use of natural language processing, it is possible to obtain additional information concerning item responses from computer programs such as e‐rater ® . In addition, available information relevant to examinee performance may include scores on related tests. We suggest application of standard results from classical test theory to the available data to obtain best linear predictors of true traditional test scores. In performing such analysis, we require estimation of variances and covariances of measurement errors, a task which can be quite difficult in the case of tests with limited numbers of items and with multiple measurements per item. As a consequence, a new estimation method is suggested based on samples of examinees who have taken an assessment more than once. Such samples are typically not random samples of the general population of examinees, so that we apply statistical adjustment methods to obtain the needed estimated variances and covariances of measurement errors. To examine practical implications of the suggested methods of analysis, applications are made to GRE G eneral A nalytical W riting and TOEFL i BT W riting. Results obtained indicate that substantial improvements are possible both in terms of reliability of scoring and in terms of assessment reliability.Keywords
This publication has 13 references indexed in Scilit:
- USE OF E‐RATER® IN SCORING OF THE TOEFL IBT® WRITING TESTETS Research Report Series, 2011
- The Application of the Cumulative Logistic Regression Model to Automated Essay ScoringJournal of Educational and Behavioral Statistics, 2010
- When Can Subscores Have Value?Journal of Educational and Behavioral Statistics, 2008
- Linear Prediction of a True Score From a Direct Estimate and Several Derived EstimatesJournal of Educational and Behavioral Statistics, 2007
- Informed Test Component WeightingEducational Measurement: Issues and Practice, 2001
- Adjustment by Minimum Discriminant InformationThe Annals of Statistics, 1984
- $I$-Divergence Geometry of Probability Distributions and Minimization ProblemsThe Annals of Probability, 1975
- Linear Statistical Inference and its Applications1973
- On Information and SufficiencyThe Annals of Mathematical Statistics, 1951
- Theory of mental tests.1950