Examiners and content and site: Oh My! A national organization’s investigation of score variation in large-scale performance assessments
- 28 August 2014
- journal article
- Published by Springer Nature in Advances in Health Sciences Education
- Vol. 20 (3) , 581-594
Abstract
Examiner effects and content specificity are two well known sources of construct irrelevant variance that present great challenges in performance-based assessments. National medical organizations that are responsible for large-scale performance based assessments experience an additional challenge as they are responsible for administering qualification examinations to physician candidates at several locations and institutions. This study explores the impact of site location as a source of score variation in a large-scale national assessment used to measure the readiness of internationally educated physician candidates for residency programs. Data from the Medical Council of Canada’s National Assessment Collaboration were analyzed using Hierarchical Linear Modeling and Rasch Analyses. Consistent with previous research, problematic variance due to examiner effects and content specificity was found. Additionally, site location was also identified as a potential source of construct irrelevant variance in examination scores.Keywords
This publication has 20 references indexed in Scilit:
- Expertise in performance assessment: assessors’ perspectivesAdvances in Health Sciences Education, 2012
- Good questions, good answers: construct alignment improves the performance of workplace‐based assessment scalesMedical Education, 2011
- Perspective: The ACGME Toolbox: Half Empty or Half Full?Academic Medicine, 2010
- Rater Errors in a Clinical Skills Assessment of Medical StudentsEvaluation & the Health Professions, 2007
- Undesired variance due to examiner stringency/leniency effect in communication skill scores assessed in OSCEsAdvances in Health Sciences Education, 2007
- Quality control of an OSCE using generalizability theory and many-faceted Rasch measurementAdvances in Health Sciences Education, 2007
- Multivariate Generalizability Analysis of the Impact of Training and Examinee Performance Information on Judgments Made in an Angoff‐Style Standard‐Setting ProcedureJournal of Educational Measurement, 2002
- Assessing Sources of Score Variability in a Multisite Medical Performance AssessmentAcademic Medicine, 2001
- Performance‐Based Assessment: Implications of Task SpecificityEducational Measurement: Issues and Practice, 1994
- Performance rating.Psychological Bulletin, 1980