Subjective vs. objective assessment of simulation performance on laparoscopic cholecystectomy: are we evaluating the right things?
Authors:
- 1 February 2022
- journal article
- research article
- Published by Springer Nature in Surgical Endoscopy
- Vol. 36 (9) , 6661-6671
Abstract
Background Simulation using virtual reality (VR) simulators is an important tool in surgical training. VR laparoscopic simulators can provide immediate objective performance assessment without observer evaluation. This study aims to explore the correlation between subjective observer evaluation and VR laparoscopic simulator performance metrics in a laparoscopic cholecystectomy (LC) simulation module. Methods A LC simulation module using a VR laparoscopic simulator was completed by PGY2–3 general surgery residents at a single institution. Simulation performance was recorded and evaluated by a trained evaluator using the validated Global Operative Assessment of Laparoscopic Skills (GOALS) form, the Objective Structured Assessment of Technical Skills (OSATS) form, and a LC-specific simulation assessment form (LC-SIM). Objective performance metrics were also obtained from the simulator system. Performance before the curriculum (pre-test) and after the curriculum (post-test) were compared. Results Fourteen residents were included in the study. There were significant improvements from pre-test to post-test on each component of GOALS, OSATS, and LC-SIM scores (all p values < 0.05). In terms of objective simulator metrics, significant improvements were noted in time to extract gallbladder (481 ± 221 vs 909 ± 366 min, p = 0.019), total number of movements (475 ± 264 vs 839 ± 324 min, p = 0.012), and total path length (955 ± 475 vs 1775 ± 632 cm, p = 0.012) from pre-test to post-test. While number of movements and total path lengths of both hands decreased, speed of right instrument also decreased from 4.1 + 2.7 to 3.0 ± 0.7 cm/sec (p = 0.007). Average speed of left instrument was associated with respect for tissue (r = 0.60, p < 0.05) and depth perception (r = 0.68, p < 0.05) on post-test evaluations. Conclusion Our study demonstrated significant improvement in technical skills based on subjective evaluator assessment as well as objective simulator metrics after simulation. The few correlations identified between the subjective evaluator and the objective simulator assessments suggest the two evaluation modalities were measuring different aspects of the technical skills and should both be considered in the evaluation process.Keywords
This publication has 11 references indexed in Scilit:
- Surgical Simulation: Markers of ProficiencyJournal of Surgical Education, 2019
- Virtual reality training for surgical trainees in laparoscopic surgeryCochrane Database of Systematic Reviews, 2013
- A Controlled Study to Determine Measurement Conditions Necessary for a Reliable and Valid Operative Performance AssessmentAnnals of Surgery, 2012
- Virtual reality simulation training for health professions trainees in gastrointestinal endoscopyCochrane Database of Systematic Reviews, 2012
- Psychomotor control in a virtual laparoscopic surgery training environment: gaze control parameters differentiate novices from expertsSurgical Endoscopy, 2010
- Development of a virtual reality training curriculum for laparoscopic cholecystectomyBritish Journal of Surgery, 2009
- Using a dynamic training environment to acquire laparoscopic surgery skillSurgical Endoscopy, 2009
- An Evidence-Based Virtual Reality Training Program for Novice Laparoscopic SurgeonsAnnals of Surgery, 2006
- A global assessment tool for evaluation of intraoperative laparoscopic skillsThe American Journal of Surgery, 2005
- Objective structured assessment of technical skill (OSATS) for surgical residentsBritish Journal of Surgery, 1997