The Effects of Coaching on Situational Judgment Tests in High‐stakes Selection
Authors:
- 6 August 2012
- journal article
- research article
- Published by Wiley in International Journal of Selection and Assessment
- Vol. 20 (3) , 272-282
Abstract
Although the evidence for the use of situational judgment tests (SJTs) in high‐stakes testing has been generally promising, questions have been raised regarding the potential coachability of SJTs. This study reports the first examination of the effects of coaching on SJT scores in an operational high‐stakes setting. We contrast findings from a simple comparison of SJT scores for coached and uncoached participants (posttest only) with three different approaches to deal with the effects of self‐selection into coaching programs, namely using a pretest as a covariate and using two different forms of propensity score‐based matching using a wide range of variables as covariates. Coaching effects were estimated at about 0.5 SDs. The implications for the use of SJTs in high‐stakes settings and for coaching research in general are discussed.Keywords
This publication has 36 references indexed in Scilit:
- Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research.Psychological Methods, 2010
- Differentiating specific job knowledge from implicit trait policies in procedural knowledge measured by a situational judgment test.Journal of Applied Psychology, 2010
- The effects of response instructions on situational judgment test performance and validity in a high-stakes context.Journal of Applied Psychology, 2009
- SITUATIONAL JUDGMENT TESTS, RESPONSE INSTRUCTIONS, AND VALIDITY: A META‐ANALYSISPersonnel Psychology, 2007
- Conditioning on the propensity score can result in biased estimation of common measures of treatment effect: a Monte Carlo studyStatistics in Medicine, 2006
- Estimating and Using Propensity Scores with Partially Missing DataJournal of the American Statistical Association, 2000
- Coaching for the Scholastic Aptitude Test: Further Synthesis and AppraisalReview of Educational Research, 1990
- Reducing Bias in Observational Studies Using Subclassification on the Propensity ScoreJournal of the American Statistical Association, 1984
- Effects of Coaching Programs on Achievement Test PerformanceReview of Educational Research, 1983
- The central role of the propensity score in observational studies for causal effectsBiometrika, 1983