Validation of a Detailed Scoring Checklist for Use During Advanced Cardiac Life Support Certification
Authors:
- Matthew D. McEvoyMMMatthew D. McEvoy
- Jeremy C. SmalleyJSJeremy C. Smalley
- Paul J. NietertPNPaul J. Nietert
- Larry C. FieldLFLarry C. Field
- Cory M. FurseCFCory M. Furse
- John W. BlenkoJBJohn W. Blenko
- Benjamin G. CobbBCBenjamin G. Cobb
- Jenna L. WaltersJWJenna L. Walters
- Allen PendarvisAPAllen Pendarvis
- Nishita S. DalalNDNishita S. Dalal
- John J. Iii SchaeferJSJohn J. Iii Schaefer
- 1 August 2012
- journal article
- research article
- Published by Wolters Kluwer Health in Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare
- Vol. 7 (4) , 222-235
Abstract
Introduction: Defining valid, reliable, defensible, and generalizable standards for the evaluation of learner performance is a key issue in assessing both baseline competence and mastery in medical education. However, before setting these standards of performance, the reliability of the scores yielding from a grading tool must be assessed. Accordingly, the purpose of this study was to assess the reliability of scores generated from a set of grading checklists used by nonexpert raters during simulations of American Heart Association (AHA) Megacodes. Methods: The reliability of scores generated from a detailed set of checklists, when used by 4 nonexpert raters, was tested by grading team leader performance in 8 Megacode scenarios. Videos of the scenarios were reviewed and rated by trained faculty facilitators and a group of nonexpert raters. The videos were reviewed “continuously” and “with pauses.” The grading made by 2 content experts served as the reference standard, and 4 nonexpert raters were used to test the reliability of the checklists. Results: Our results demonstrate that nonexpert raters are able to produce reliable grades when using the checklists under consideration, demonstrating excellent intrarater reliability and agreement with a reference standard. The results also demonstrate that nonexpert raters can be trained in the proper use of the checklist in a short amount of time, with no discernible learning curve thereafter. Finally, our results show that a single trained rater can achieve reliable scores of team leader performance during AHA Megacodes when using our checklist in a continuous mode because measures of agreement in total scoring were very strong [Lin’s (Biometrics 1989;45:255–268) concordance correlation coefficient, 0.96; intraclass correlation coefficient, 0.97]. Conclusions: We have shown that our checklists can yield reliable scores, are appropriate for use by nonexpert raters, and are able to be used during continuous assessment of team leader performance during the review of a simulated Megacode. This checklist may be more appropriate for use by advanced cardiac life support instructors during Megacode assessments than the current tools provided by the AHA.Keywords
This publication has 24 references indexed in Scilit:
- Using Standardized Patients to Assess Professionalism: A Generalizability StudyTeaching and Learning in Medicine, 2010
- Delayed Time to Defibrillation after Intraoperative and Periprocedural Cardiac ArrestAnesthesiology, 2010
- Using Standardized Patients to Assess the Communication Skills of Graduating Physicians for the Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 2-Performance Evaluation (Level 2-PE)Teaching and Learning in Medicine, 2010
- Setting Defensible Standards for Cardiac Auscultation Skills in Medical StudentsAcademic Medicine, 2009
- Development and Validation of a Cardiac Findings Checklist for Use With Simulator-Based Assessments of Cardiac Physical Examination CompetenceSimulation in Healthcare: The Journal of the Society for Simulation in Healthcare, 2009
- Professionalism and Communication in the Intensive Care UnitSimulation in Healthcare: The Journal of the Society for Simulation in Healthcare, 2008
- Simulation performance checklist generation using the Delphi techniqueCanadian Journal of Anesthesia/Journal canadien d'anesthésie, 2007
- Mastery learning of advanced cardiac life support skills by internal medicine residents using simulation technology and deliberate practiceJournal of General Internal Medicine, 2006
- Graduating internal medicine residents’ self-assessment and performance of advanced cardiac life support skillsMedical Teacher, 2006
- Generalizability theory.American Psychologist, 1989