Assessment of Robotic Console Skills (ARCS): construct validity of a novel global rating scale for technical skills in robotically assisted surgery
- 1 July 2017
- journal article
- research article
- Published by Springer Nature in Surgical Endoscopy
- Vol. 32 (1) , 526-535
Abstract
Background Skill assessment during robotically assisted surgery remains challenging. While the popularity of the Global Evaluative Assessment of Robotics Skills (GEARS) has grown, its lack of discrimination between independent console skills limits its usefulness. The purpose of this study was to evaluate construct validity and interrater reliability of a novel assessment designed to overcome this limitation. Methods We created the Assessment of Robotic Console Skills (ARCS), a global rating scale with six console skill domains. Fifteen volunteers who were console surgeons for 0 (“novice”), 1–100 (“intermediate”), or >100 (“experienced”) robotically assisted procedures performed three standardized tasks. Three blinded raters scored the task videos using ARCS, with a 5-point Likert scale for each skill domain. Scores were analyzed for evidence of construct validity and interrater reliability. Results Group demographics were indistinguishable except for the number of robotically assisted procedures performed (p = 0.001). The mean scores of experienced subjects exceeded those of novices in dexterity (3.8 > 1.4, p < 0.001), field of view (4.1 > 1.8, p < 0.001), instrument visualization (3.9 > 2.2, p < 0.001), manipulator workspace (3.6 > 1.9, p = 0.001), and force sensitivity (4.3 > 2.6, p < 0.001). The mean scores of intermediate subjects exceeded those of novices in dexterity (2.8 > 1.4, p = 0.002), field of view (2.8 > 1.8, p = 0.021), instrument visualization (3.2 > 2.2, p = 0.045), manipulator workspace (3.1 > 1.9, p = 0.004), and force sensitivity (3.7 > 2.6, p = 0.033). The mean scores of experienced subjects exceeded those of intermediates in dexterity (3.8 > 2.8, p = 0.003), field of view (4.1 > 2.8, p < 0.001), and instrument visualization (3.9 > 3.2, p = 0.044). Rater agreement in each domain demonstrated statistically significant concordance (p < 0.05). Conclusions We present strong evidence for construct validity and interrater reliability of ARCS. Our study shows that learning curves for some console skills plateau faster than others. Therefore, ARCS may be more useful than GEARS to evaluate distinct console skills. Future studies will examine why some domains did not adequately differentiate between subjects and applications for intraoperative use.Keywords
Funding Information
- Intuitive Surgical (Employer funding)
This publication has 30 references indexed in Scilit:
- Comparative assessment of three standardized robotic surgery training methodsBJU International, 2013
- Developing a comprehensive, proficiency-based training program for robotic surgerySurgery, 2012
- Objective measures for longitudinal assessment of robotic surgery trainingThe Journal of Thoracic and Cardiovascular Surgery, 2012
- Assessing system operation skills in robotic surgery traineesThe International Journal of Medical Robotics and Computer Assisted Surgery, 2011
- What is the Learning Curve for Robotic Assisted Gynecologic Surgery?Journal of Minimally Invasive Gynecology, 2008
- Towards automatic skill evaluation: Detection and segmentation of robot-assisted surgical motionsComputer Aided Surgery, 2006
- Face validity of assessments: faith‐based interpretations or evidence‐based science?Medical Education, 2005
- A global assessment tool for evaluation of intraoperative laparoscopic skillsThe American Journal of Surgery, 2005
- Dexterity enhancement with robotic surgerySurgical Endoscopy, 2004
- Testing technical skill via an innovative “bench station” examinationThe American Journal of Surgery, 1997