Simulation-based Assessment to Identify Critical Gaps in Safe Anesthesia Resident Performance
Authors:
- Richard H. BlumRBRichard H. Blum
- John R. BouletJBJohn R. Boulet
- Jeffrey B. CooperJCJeffrey B. Cooper
- Sharon L. Muret-WagstaffSMSharon L. Muret-Wagstaff
Open Access
- 1 January 2014
- journal article
- research article
- Published by Wolters Kluwer Health in Anesthesiology
- Vol. 120 (1) , 129-141
Abstract
Background: Valid methods are needed to identify anesthesia resident performance gaps early in training. However, many assessment tools in medicine have not been properly validated. The authors designed and tested use of a behaviorally anchored scale, as part of a multiscenario simulation-based assessment system, to identify high- and low-performing residents with regard to domains of greatest concern to expert anesthesiology faculty. Methods: An expert faculty panel derived five key behavioral domains of interest by using a Delphi process (1) Synthesizes information to formulate a clear anesthetic plan; (2) Implements a plan based on changing conditions; (3) Demonstrates effective interpersonal and communication skills with patients and staff; (4) Identifies ways to improve performance; and (5) Recognizes own limits. Seven simulation scenarios spanning pre-to-postoperative encounters were used to assess performances of 22 first-year residents and 8 fellows from two institutions. Two of 10 trained faculty raters blinded to trainee program and training level scored each performance independently by using a behaviorally anchored rating scale. Residents, fellows, facilitators, and raters completed surveys. Results: Evidence supporting the reliability and validity of the assessment scores was procured, including a high generalizability coefficient (ρ2 = 0.81) and expected performance differences between first-year resident and fellow participants. A majority of trainees, facilitators, and raters judged the assessment to be useful, realistic, and representative of critical skills required for safe practice. Conclusion: The study provides initial evidence to support the validity of a simulation-based performance assessment system for identifying critical gaps in safe anesthesia resident performance early in training.Keywords
This publication has 34 references indexed in Scilit:
- The Next GME Accreditation System — Rationale and BenefitsNew England Journal of Medicine, 2012
- What should be included in a simulation course for anaesthetists? The Merseyside trainee perspectiveEuropean Journal of Anaesthesiology, 2012
- Simulation-based Assessment of Pediatric Anesthesia SkillsAnesthesiology, 2011
- Criteria for good assessment: Consensus statement and recommendations from the Ottawa 2010 ConferenceMedical Teacher, 2011
- Comparison of Checklist and Anchored Global Rating Instruments for Performance Rating of Simulated Pediatric EmergenciesSimulation in Healthcare: The Journal of the Society for Simulation in Healthcare, 2011
- Objective Structured Clinical Examination–Based Assessment of Regional Anesthesia SkillsAnesthesia & Analgesia, 2011
- Performance of Certified Registered Nurse Anesthetists and Anesthesiologists in a Simulation-Based Skills AssessmentAnesthesia & Analgesia, 2009
- Setting Performance Standards for Mannequin-Based Acute-Care ScenariosSimulation in Healthcare: The Journal of the Society for Simulation in Healthcare, 2008
- Using Standardized Patients to Assess the Interpersonal Skills of Physicians: Six Years' Experience With a High-Stakes Certification ExaminationHealth Communication, 2007
- Role of Experience in the Response to Simulated Critical IncidentsAnesthesia & Analgesia, 1991