Going beyond hawks and doves – Measuring degrees of examiner misalignment in OSCEs
Author:
Open Access
- 5 February 2025
- journal article
- Published by Taylor & Francis in Medical Teacher
- Vol. 47 (10) , 1630-1636
Abstract
Minimising examiner differences in scoring in OSCEs is key in supporting the validity of the assessment outcomes. This is particularly true for common OSCE designs where the same station is administered across parallel circuits, with examiners nested within these. However, the common classification of extreme examiners as 'hawks' or 'doves' can be overly simplistic. Rather, it is the difference in patterns of scoring across circuits that better indicates poor levels of agreement between examiners that can unfairly advantage particular groups of candidates in comparison with others in different circuits. In this paper, a new measure of differences in examiner scoring is presented that quantifies the different combined patterns of scoring in global grades and checklist/domain scores for pairs of examiners assessing in the same station but in different circuits. Based on calculating the area between separate examiners' individual borderline regression lines, this measure can be used as a post hoc metric to provide a broad range of validity evidence for the assessment and its outcomes. In challenging the 'hawks'/'doves' paradigm, this work presents a detailed empirical analysis of a new misalignment metric in a particular high-stakes context and gives a range of evidence of its contribution to overall OSCE quality control processes and of improved fairness to candidates over time. The paper concludes with comments on developing the metric to contexts where there are multiple parallel circuits which will allows its practical application to a broader set of OSCE contexts.Keywords
This publication has 12 references indexed in Scilit:
- Inter-school variations in the standard of examiners’ graduation-level OSCE judgementsMedical Teacher, 2024
- Towards a more nuanced conceptualisation of differential examiner stringency in OSCEsAdvances in Health Sciences Education, 2023
- Robust, defensible, and fair: The AMEE guide to selection into medical school: AMEE Guide No. 153Medical Teacher, 2023
- Pass/fail decisions and standards: the impact of differential examiner stringency on OSCE outcomesAdvances in Health Sciences Education, 2022
- Developing a video‐based method to compare and adjust examiner effects in fully nested OSCEsMedical Education, 2018
- Examiners and content and site: Oh My! A national organization’s investigation of score variation in large-scale performance assessmentsAdvances in Health Sciences Education, 2014
- The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part I: An historical and theoretical perspectiveMedical Teacher, 2013
- Longitudinal interrelationships of OSCE station level analyses, quality improvement and overall reliabilityMedical Teacher, 2013
- How to measure the quality of the OSCE: A review of metrics – AMEE guide no. 49Medical Teacher, 2010
- Comparison of a rational and an empirical standard setting procedure for an OSCEMedical Education, 2003