C-SATS: Assessing Surgical Skills Among Urology Residency Applicants
- 1 April 2017
- journal article
- research article
- Published by SAGE Publications in Journal of Endourology
- Vol. 31 (S1) , S95-95
Abstract
Background: We hypothesized that surgical skills assessment could aid in the selection process of medical student applicants to a surgical program. Recently, crowdsourcing has been shown to provide an accurate assessment of surgical skills at all levels of training. We compared expert and crowd assessment of surgical tasks performed by resident applicants during their interview day at the urology program at the University of California, Irvine. Materials and Methods: Twenty-five resident interviewees performed four tasks: open square knot tying, laparoscopic peg transfer, robotic suturing, and skill task 8 on the LAP Mentor™ (Simbionix Ltd., Lod, Israel). Faculty experts and crowd workers (Crowd-Sourced Assessment of Technical Skills [C-SATS], Seattle, WA) assessed recorded performances using the Objective Structured Assessment of Technical Skills (OSATS), Global Evaluative Assessment of Robotic Skills (GEARS), and the Global Operative Assessment of Laparoscopic Skills (GOALS) validated assessment tools. Results: Overall, 3938 crowd assessments were obtained for the four tasks in less than 3.5 hours, whereas the average time to receive 150 expert assessments was 22 days. Inter-rater agreement between expert and crowd assessment scores was 0.62 for open knot tying, 0.92 for laparoscopic peg transfer, and 0.86 for robotic suturing. Agreement between applicant rank on skill task 8 on the LAP Mentor assessment and crowd assessment was 0.32. The crowd match rank based solely on skills performance did not compare well with the final faculty match rank list (0.46); however, none of the bottom five crowd-rated applicants appeared in the top five expert-rated applicants and none of the top five crowd-rated applicants appeared in the bottom five expert-rated applicants. Conclusions: Crowd-source assessment of resident applicant surgical skills has good inter-rater agreement with expert physician raters but not with a computer-based objective motion metrics software assessment. Overall applicant rank was affected to some degree by the crowd performance rating.Keywords
This publication has 19 references indexed in Scilit:
- Crowdsourcing: a valid alternative to expert evaluation of robotic surgery skillsAmerican Journal of Obstetrics and Gynecology, 2016
- Crowd-Sourced Assessment of Technical Skills for Validation of Basic Laparoscopic Urologic Skills TasksJournal of Urology, 2016
- Crowd-Sourced Assessment of Technical Skill: A Valid Method for Discriminating Basic Robotic Surgery SkillsJournal of Endourology, 2015
- Crowd-Sourced Assessment of Technical Skills: Differentiating Animate Surgical Skill Through the Wisdom of CrowdsJournal of Endourology, 2015
- Crowd-Sourced Assessment of Technical Skills: An Adjunct to Urology Resident Surgical Simulation TrainingJournal of Endourology, 2015
- Crowd-Sourced Assessment of Technical Skills: a novel method to evaluate surgical performanceJournal of Surgical Research, 2014
- Surgical Skill and Complication Rates after Bariatric SurgeryNew England Journal of Medicine, 2013
- Positive Correlation Between Motion Analysis Data on the LapMentor Virtual Reality Laparoscopic Surgical Simulator and the Results from Videotape Assessment of Real Laparoscopic SurgeriesJournal of Endourology, 2012
- The Endoscopic Surgical Skill Qualification System in Urological Laparoscopy: A Novel System in JapanJournal of Urology, 2006
- Evaluating surgical competency with the American Board of Surgery In-Training Examination, skill testing, and intraoperative assessmentSurgery, 2000