Construct Validity and Generalizability of Simulation-Based Objective Structured Clinical Examination Scenarios
Authors:
- Avner SidiASAvner Sidi
- Nikolaus GravensteinNGNikolaus Gravenstein
- Samsun LampotangSLSamsun Lampotang
- 1 September 2014
- journal article
- Published by Journal of Graduate Medical Education in Journal of Graduate Medical Education
- Vol. 6 (3) , 489-494
Abstract
Background: It is not known if construct-related validity (progression of scores with different levels of training) and generalizability of Objective Structured Clinical Examination (OSCE) scenarios previously used with non-US graduating anesthesiology residents translate to a US training program. Objective: We assessed for progression of scores with training for a validated high-stakes simulation-based anesthesiology examination. Methods: Fifty US anesthesiology residents in postgraduate years (PGYs) 2 to 4 were evaluated in operating room, trauma, and resuscitation scenarios developed for and used in a high-stakes Israeli Anesthesiology Board examination, requiring a score of 70% on the checklist for passing (including all critical items). Results: The OSCE error rate was lower for PGY-4 than PGY-2 residents in each field, and for most scenarios within each field. The critical item error rate was significantly lower for PGY-4 than PGY-3 residents in operating room scenarios, and for PGY-4 than PGY-2 residents in resuscitation scenarios. The final pass rate was significantly higher for PGY-3 and PGY-4 than PGY-2 residents in operating room scenarios, and also was significantly higher for PGY-4 than PGY-2 residents overall. PGY-4 residents had a better error rate, total scenarios score, general evaluation score, critical items error rate, and final pass rate than PGY-2 residents. Conclusions: The comparable error rates, performance grades, and pass rates for US PGY-4 and non-US (Israeli) graduating (PGY-4 equivalent) residents, and the progression of scores among US residents with training level, demonstrate the construct-related validity and generalizability of these high-stakes OSCE scenarios.Keywords
This publication has 17 references indexed in Scilit:
- Simulation-based Assessment in AnesthesiologyAnesthesiology, 2010
- Credentialing and Certifying with SimulationAnesthesiology Clinics, 2007
- Incorporating Simulation-Based Objective Structured Clinical Examination into the Israeli National Board Examination in AnesthesiologyAnesthesia & Analgesia, 2006
- The Feasibility of Sharing Simulation-Based Evaluation Scenarios in AnesthesiologyAnesthesia & Analgesia, 2005
- The Step 2 Clinical-Skills ExaminationNew England Journal of Medicine, 2004
- Crisis resource management training for an anaesthesia faculty: a new approach to continuing educationMedical Education, 2004
- Evaluation of Anesthesia Residents Using Mannequin-based SimulationAnesthesiology, 2002
- Equivalence Testing the Easy WayJournal of Computing and Information Technology, 2002
- Testing Internal Consistency and Construct Validity During Evaluation of Performance in a Patient SimulatorAnesthesia & Analgesia, 1998
- Testing Internal Consistency and Construct Validity During Evaluation of Performance in a Patient SimulatorAnesthesia & Analgesia, 1998