Exploring the impact of mental workload on rater-based assessments
Authors:
- 7 April 2012
- journal article
- review article
- Published by Springer Nature in Advances in Health Sciences Education
- Vol. 18 (2) , 291-303
Abstract
When appraising the performance of others, assessors must acquire relevant information and process it in a meaningful way in order to translate it effectively into ratings, comments, or judgments about how well the performance meets appropriate standards. Rater-based assessment strategies in health professional education, including scale and faculty development strategies aimed at improving them have generally been implemented with limited consideration of human cognitive and perceptual limitations. However, the extent to which the task assigned to raters aligns with their cognitive and perceptual capacities will determine the extent to which reliance on human judgment threatens assessment quality. It is well recognized in medical decision making that, as the amount of information to be processed increases, judges may engage mental shortcuts through the application of schemas, heuristics, or the adoption of solutions that satisfy rather than optimize the judge’s needs. Further, these shortcuts may fundamentally limit/bias the information perceived or processed. Thinking of the challenges inherent in rater-based assessments in an analogous way may yield novel insights regarding the limits of rater-based assessment and may point to greater understanding of ways in which raters can be supported to facilitate sound judgment. This paper presents an initial exploration of various cognitive and perceptual limitations associated with rater-based assessment tasks. We hope to highlight how the inherent cognitive architecture of raters might beneficially be taken into account when designing rater-based assessment protocols.Keywords
This publication has 64 references indexed in Scilit:
- Workplace-based assessment: effects of rater expertiseAdvances in Health Sciences Education, 2010
- Applying the science of learning to medical educationMedical Education, 2010
- Internal structure of mini-CEX scores for internal medicine residents: factor analysis and generalizabilityAdvances in Health Sciences Education, 2010
- Decision making and the avoidance of cognitive demand.Journal of Experimental Psychology: General, 2010
- Anticipation of cognitive demand during decision-makingPsychological Research, 2008
- Use of the Mini-Clinical Evaluation Exercise to Rate Examinee Performance on a Multiple-Station Clinical Skills Examination: A Validity StudyAcademic Medicine, 2006
- Medical School Admissions: Enhancing the Reliability and Validity of an Autobiographical Screening ToolAcademic Medicine, 2006
- Inattentional BlindnessCurrent Directions in Psychological Science, 2003
- Performance Appraisal Process Research in the 1980s: What Has It Contributed to Appraisals in Use?Organizational Behavior and Human Decision Processes, 1993
- Ubiquitous halo.Psychological Bulletin, 1981