Investigating disparity between global grades and checklist scores in OSCEs
Authors:
- 16 February 2015
- journal article
- research article
- Published by Taylor & Francis in Medical Teacher
- Vol. 37 (12) , 1106-1113
Abstract
Background: When measuring assessment quality, increasing focus is placed on the value of station-level metrics in the detection and remediation of problems in the assessment. Aims: This article investigates how disparity between checklist scores and global grades in an Objective Structured Clinical Examination (OSCE) can provide powerful new insights at the station level whenever such disparities occur and develops metrics to indicate when this is a problem. Method: This retrospective study uses OSCE data from multiple examinations to investigate the extent to which these new measurements of disparity complement existing station-level metrics. Results: In stations where existing metrics are poor, the new metrics provide greater understanding of the underlying sources of error. Equally importantly, stations of apparently satisfactory “quality” based on traditional metrics are shown to sometimes have problems of their own – with a tendency for checklist score “performance” to be judged stronger than would be expected from the global grades awarded. Conclusions: There is an ongoing tension in OSCE assessment between global holistic judgements and the necessarily more reductionist, but arguably more objective, checklist scores. This article develops methods to quantify the disparity between these judgements and illustrates how such analyses can inform ongoing improvement in station quality.Keywords
This publication has 16 references indexed in Scilit:
- Rater-Based Assessments as Social Judgments: Rethinking the Etiology of Rater ErrorsAcademic Medicine, 2011
- Opening the black box of clinical skills assessment via observation: a conceptual modelMedical Education, 2011
- How to measure the quality of the OSCE: A review of metrics – AMEE guide no. 49Medical Teacher, 2010
- Fidelity as a precondition for integrity in grading academic achievementAssessment & Evaluation in Higher Education, 2009
- Indeterminacy in the use of preset criteria for assessment and gradingAssessment & Evaluation in Higher Education, 2009
- Broadening Perspectives on Clinical Performance Assessment: Rethinking the Nature of In-training AssessmentAdvances in Health Sciences Education, 2006
- Setting standards for student assessmentInternational Journal of Research & Method in Education, 2006
- Comparison of a rational and an empirical standard setting procedure for an OSCEMedical Education, 2003
- Standard SettingAcademic Medicine, 2002
- AMEE Guide No. 18: Standard setting in student assessmentMedical Teacher, 2000