Interrater reliability and agreement of the NEUMOBACT checklist about infection-prevention performance of intensive care nurses in simulation-based scenarios
Open Access
- 31 December 2024
- journal article
- research article
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 19 (12) , e0313175
Abstract
Objective: To analyse the interrater reliability of the NEUMOBACT checklist and verify whether consistent results are reproducible. Methods: A validation study with a cross-sectional design, compliant with the GRRAS checklist, among ICU nurses attending a SIMULAZERO course with an Objective Structured Clinical Evaluation simulation format, to verify transfer from theory to clinical practice of knowledge and skills in ventilator-associated pneumonia (VAP) and catheter-related bacteraemia (CRB) prevention. A minimum sample size of 111 pairs of nurse raters was calculated. Interrater agreement was analysed using Gwet’s AC1 for each item and as a total for each of the three checklists in the NEUMOBACT instrument. Results: A total of 95 pairs of valid NEUMOBACT checklists were completed by 190 raters with a median age of 29 [25–35] years, 93.7% were female. At the Central Venous Catheter insertion station, Gwet’s AC1 was 0.934 (95% CI [0.919–0.949]). Only 2 of the 17 items scored below 0.9. At the Endotracheal Suctioning station, Gwet’s AC1 was 0.869 (95% CI [0.851–0.886]). Of the 26 items that made up this station, 16 had an agreement percentage above 0.9, a further 9 were between 0.821 and 0.884, and item 13 had a value of 0.789. At the Patient Care station, Gwet’s AC1 was 0.911 (95% CI [0.896–0.927]). Of the 21 items, 17 showed an agreement percentage above 0.9 and 4 were between 0.810 and 0.894. Conclusions: The interrater reliability of the NEUMOBACT checklist shows substantial agreement between pairs of raters and is therefore validated in this large sample of ICU nurses. Relevance to clinical practice: The NEUMOBACT checklist can be useful for assessing skills before and after training in VAP and CRB prevention measures and during debriefing (post-simulation feedback), to reinforce the scientific evidence behind actions and decisions for items that have been performed incorrectly, thus consolidating training already received.Keywords
Funding Information
- SESSEP Research Grant; https://www.sessep.com/becas-sessep/ (Research Grant 2021)
This publication has 27 references indexed in Scilit:
- Value of Nursing Objective Structured Clinical ExaminationsNurse Educator, 2019
- Impact of an education program on the performance of nurses in providing oral care for mechanically ventilated childrenAustralian Critical Care, 2019
- Reliability and validity study of the Spanish adaptation of the “Creighton Simulation Evaluation Instrument (C-SEI)”Nurse Education in Practice, 2019
- Effects of simulation education on oral care practices – a randomized controlled trialNursing in Critical Care, 2017
- The reliability and validity of three questionnaires: The Student Satisfaction and Self-Confidence in Learning Scale, Simulation Design Scale, and Educational Practices QuestionnaireContemporary Nurse, 2016
- Simulation-Based Mastery Learning Improves Central Line Maintenance Skills of ICU NursesJONA: The Journal of Nursing Administration, 2015
- Development and psychometric testing of ventilator bundle questionnaire and observation scheduleAmerican Journal of Infection Control, 2014
- Simulation-Based Training for Nurses in Sterile Techniques During Central Vein CatheterizationAmerican Journal of Critical Care, 2014
- A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samplesBMC Medical Research Methodology, 2013
- Assessment of a training programme for the prevention of ventilator‐associated pneumoniaNursing in Critical Care, 2012