Reliability of team-based self-monitoring in critical events: a pilot study
Open Access
- 1 December 2013
- journal article
- research article
- Published by Springer Nature in BMC Emergency Medicine
- Vol. 13 (1) , 22
Abstract
Teamwork is a critical component during critical events. Assessment is mandatory for remediation and to target training programmes for observed performance gaps. The primary purpose was to test the feasibility of team-based self-monitoring of crisis resource management with a validated teamwork assessment tool. A secondary purpose was to assess item-specific reliability and content validity in order to develop a modified context-optimised assessment tool. We conducted a prospective, single-centre study to assess team-based self-monitoring of teamwork after in-situ inter-professional simulated critical events by comparison with an assessment by observers. The Mayo High Performance Teamwork Scale (MHPTS) was used as the assessment tool with evaluation of internal consistency, item-specific consensus estimates for agreement between participating teams and observers, and content validity. 105 participants and 58 observers completed the MHPTS after a total of 16 simulated critical events over 8 months. Summative internal consistency of the MHPTS calculated as Cronbach’s alpha was acceptable with 0.712 for observers and 0.710 for participants. Overall consensus estimates for dichotomous data (agreement/non-agreement) was 0.62 (Cohen’s kappa; IQ-range 0.31-0.87). 6/16 items had excellent (kappa > 0.8) and 3/16 good reliability (kappa > 0.6). Short questions concerning easy to observe behaviours were more likely to be reliable. The MHPTS was modified using a threshold for good reliability of kappa > 0.6. The result is a 9 item self-assessment tool (TeamMonitor) with a calculated median kappa of 0.86 (IQ-range: 0.67-1.0) and good content validity. Team-based self-monitoring with the MHPTS to assess team performance during simulated critical events is feasible. A context-based modification of the tool is achievable with good internal consistency and content validity. Further studies are needed to investigate if team-based self-monitoring may be used as part of a programme of assessment to target training programmes for observed performance gaps.Keywords
This publication has 34 references indexed in Scilit:
- Simulation-based crisis resource management training for pediatric critical care medicinePediatric Critical Care Medicine, 2012
- Measuring Teamwork in Health Care Settings: A Review of Survey InstrumentsSSRN Electronic Journal, 2012
- Observational Skill-based Clinical Assessment tool for Resuscitation (OSCAR): Development and validationResuscitation, 2011
- Exploring the divergence between self-assessment and self-monitoringAdvances in Health Sciences Education, 2010
- Simulation-based training delivered directly to the pediatric cardiac intensive care unit engenders preparedness, comfort, and decreased anxiety among multidisciplinary resuscitation teamsThe Journal of Thoracic and Cardiovascular Surgery, 2010
- Simulation in Medical Education2010
- Identifying non-technical skills and barriers for improvement of teamwork in cardiac arrest teamsResuscitation, 2010
- Likert scales, levels of measurement and the “laws” of statisticsAdvances in Health Sciences Education, 2010
- Team Behavior During Trauma Resuscitation: A Simulation-Based Performance AssessmentJournal of Graduate Medical Education, 2009
- Preventing Errors in Clinical Practice: A Call for Self-AwarenessThe Annals of Family Medicine, 2004