An Empirical Approach to Identifying Subject Matter Experts for the Development of Situational Judgment Tests
Authors:
- 1 October 2021
- journal article
- research article
- Published by Hogrefe Publishing Group in Journal of Personnel Psychology
- Vol. 20 (4) , 151-163
Abstract
The development of a scoring key for the situational judgment test often requires subject matter experts (SMEs) to identify the best responses for a hypothetical situation. And yet, there is no gold standard for identifying the SMEs. This paper describes an empirical and context-free approach: the Cochran–Weiss–Shanteau (CWS) method, which does not rely on external criteria such as tenure or credential. We first describe the theory behind the empirical approach of expertise. We also outline the CWS method and provide an R script for calculating the CWS index. Next, we demonstrate how the CWS index can be used for improving interrater agreement and the efficiency of SME selection. Finally, we examined the nomological network of the CWS index. We found that the CWS index was associated with reflective thinking and intuition avoidance.Keywords
All Related Versions
This publication has 35 references indexed in Scilit:
- SITUATIONAL JUDGMENT TESTS: CONSTRUCTS ASSESSED AND A META-ANALYSIS OF THEIR CRITERION-RELATED VALIDITIESPersonnel Psychology, 2010
- Measuring Expertise in Weather-Related Aeronautical Risk Perception: The Validity of the Cochran–Weiss–Shanteau (CWS) IndexThe International Journal of Aviation Psychology, 2009
- Subgroup Differences in Situational Judgment Test Performance: A Meta-AnalysisHuman Performance, 2008
- The Cochran–Weiss–Shanteau performance index as an indicator of upper limb risk assessment expertiseErgonomics, 2008
- Situational judgment tests: a review of recent researchPersonnel Review, 2008
- People who judge peopleJournal of Behavioral Decision Making, 2006
- Scoring Situational Judgment Tests: Once You Get the Data, Your Troubles BeginInternational Journal of Selection and Assessment, 2006
- Development and Validation of a Situational Judgment Test of Employee IntegrityInternational Journal of Selection and Assessment, 2005
- Individual differences in intuitive-experiential and analytical-rational thinking styles.Journal of Personality and Social Psychology, 1996
- Personal attributes of expert auditorsOrganizational Behavior and Human Decision Processes, 1992