Remote and onsite scoring of OSCEs using generalisability theory: A three-year cohort study
Open Access
- 20 November 2018
- journal article
- research article
- Published by Taylor & Francis in Medical Teacher
- Vol. 41 (5) , 578-583
Abstract
Introduction: Onsite scoring is common in traditional OSCEs although there is the potential for an audience effect facilitating or inhibiting performance. We aim to (1) analyze the reliability between onsite scoring (OS) and remote scoring (RS); and (2) explore the factors that affect the scoring in different locations. Methods: A total of 154 students and 84 raters were enrolled in a single-site during 2013–2015. We selected six stations randomly from a 12-station national high-stakes OSCE. We applied generalisability theory for the analysis and investigated the perceptions that affected RS scoring. Results: The internal consistency reliability Cronbach’s α of the checklists was 0.92. The kappa agreement was 0.623 and the G value was 0.93. The major source of variance comes from the students themselves, but some from locations and raters. The three-component analysis including Technical Feasibility, Facilitates Wellbeing, and Observational and Attention Deficits explained 73.886% of the total variance in RS scoring. Conclusions: Our study has demonstrated moderate agreement and good reliability between OS and RS ratings. We validated the factors of facility operation and quality for RS raters. Remote scoring can provide an alternative forum for the raters to overcome the barriers of distance, space, and avoid the audience effect.Keywords
Funding Information
- Kaohsiung Chang Gung Memorial Hospital
- Chang Gung University
This publication has 17 references indexed in Scilit:
- A systematic review of the reliability of objective structured clinical examination scoresMedical Education, 2011
- Assessing Intern Core Competencies With an Objective Structured Clinical ExaminationJournal of Graduate Medical Education, 2009
- Rater Errors in a Clinical Skills Assessment of Medical StudentsEvaluation & the Health Professions, 2007
- Assisting international medical graduates applying for their first post in the UK: what should be done?Clinical Medicine, 2005
- Simulations in the United States Medical Licensing ExaminationTM (USMLETM)Quality and Safety in Health Care, 2004
- Social facilitation from Triplett to electronic performance monitoring.Group Dynamics: Theory, Research, and Practice, 2001
- Large-scale high-stakes testing with an OSCEAcademic Medicine, 1996
- The assessment of clinical skills/competence/performanceAcademic Medicine, 1990
- Social facilitation: A meta-analysis of 241 studies.Psychological Bulletin, 1983
- Effects of audience status, evaluation, and time of action on performance with hidden-word problems.Journal of Personality and Social Psychology, 1973