Modeling Reliability and Validity Across OSCE Station Numbers in Final MBBS Exam
Authors:
- Alok KumarAKAlok Kumar
- Kandamaran KrishnamurthyKKKandamaran Krishnamurthy
- Shastri MotilalSMShastri Motilal
- Michael H CampbellMCMichael H Campbell
- Joanne Paul-CharlesJPJoanne Paul-Charles
- Maritza FernandesMFMaritza Fernandes
- Kenneth ConnellKenneth Connell
- Euclid MorrisEMEuclid Morris
- Maisha EmmanuelMEMaisha Emmanuel
- Bidyadhar SaBSBidyadhar Sa
- Md Anwarul Azim MajumderMd Anwarul Azim Majumder
Open Access
- 14 August 2026
- journal article
- research article
- Published by MDPI AG in International Medical Education
- Vol. 5 (3) , 82
Abstract
Background: The Objective Structured Clinical Examination (OSCE) has been the backbone of assessment in medical education, and the number of stations required to achieve sufficient reliability in high-stakes examinations is an important consideration for resource-limited programs. Methods: The correlation between the number of stations and psychometric performance was simulated using post hoc resampling of the 17-station Final MBBS Medicine and Therapeutics OSCE across three campuses of the University of the West Indies. Blueprint coverage was maintained in balanced subsets of six, eight, ten, and twelve stations. For subsets and the full exam, we compared failure rates, internal consistency, generalizability, and correlation of scores. Results: Failure rates increased as the number of stations decreased; however, none of the differences between subset and full-exam performance were significant. Station count was positively related to internal consistency using alpha (α ≈ 0.32–0.61 at six stations; α ≈ 0.64–0.70 at 17 stations) and G-coefficients (≈0.18–0.42 at six and ≈0.36–0.70 at 17 stations). However, all reliability coefficients were <0.80. Strong internal consistency was maintained for all subsets, even at six stations (r ≥ 0.79), and approached 1.0 for the 17-station subset. There was a subset size effect on mean values for Campuses 1 and 2 but not for Campus 3. Conclusions: Shorter subset exams maintained strong internal consistency. Reliability at ten–twelve stations approached that of the 17-station total. Intercampus variability highlighted the need for examiner training, station quality checks, and improved standard setting.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Exam blueprinting as a tool to overcome principal validity threats: A scoping reviewEducación Médica, 2024
- Rapid Blueprinting: An Efficient Method for Designing Content of AssessmentsMedical Science Educator, 2024
- Setting defensible minimum-stations-passed standards in OSCE-type assessmentsMedical Teacher, 2023
- Conjunctive standards in OSCEs: The why and the how of number of stations passed criteriaMedical Teacher, 2020
- Remote and onsite scoring of OSCEs using generalisability theory: A three-year cohort studyMedical Teacher, 2018
- A systematic review of the reliability of objective structured clinical examination scoresMedical Education, 2011
- Validity: on the meaningful interpretation of assessment dataMedical Education, 2003
- The assessment of professional competence: Developments, research and practical implicationsAdvances in Health Sciences Education, 1996
- VALIDITY OF PSYCHOLOGICAL ASSESSMENT: VALIDATION OF INFERENCES FROM PERSONS' RESPONSES AND PERFORMANCES AS SCIENTIFIC INQUIRY INTO SCORE MEANINGETS Research Report Series, 1994
- THEORY OF GENERALIZABILITY: A LIBERALIZATION OF RELIABILITY THEORY†British Journal of Statistical Psychology, 1963