Association between leniency of anesthesiologists when evaluating certified registered nurse anesthetists and when evaluating didactic lectures
- 18 September 2020
- journal article
- research article
- Published by Springer Nature in Health Care Management Science
- Vol. 23 (4) , 640-648
Abstract
Daily evaluations of certified registered nurse anesthetists’ (CRNAs’) work habits by anesthesiologists should be adjusted for rater leniency. The current study tested the hypothesis that there is a pairwise association by rater between leniencies of evaluations of CRNAs’ daily work habits and of didactic lectures. The historical cohorts were anesthesiologists’ evaluations over 53 months of CRNAs’ daily work habits and 65 months of didactic lectures by visiting professors and faculty. The binary endpoints were the Likert scale scores for all 6 and 10 items, respectively, equaling the maximums of 5 for all items, or not. Mixed effects logistic regression estimated the odds of each ratee performing above or below average adjusted for rater leniency. Bivariate errors in variables least squares linear regression estimated the association between the leniency of the anesthesiologists’ evaluations of work habits and didactic lectures. There were 29/107 (27%) raters who were more severe in their evaluations of CRNAs’ work habits than other anesthesiologists (two-sided P < 0.01); 34/107 (32%) raters were more lenient. When evaluating lectures, 3/81 (4%) raters were more severe and 8/81 (10%) more lenient. Among the 67 anesthesiologists rating both, leniency (or severity) for work habits was not associated with that for lectures (P = 0.90, unitless slope between logits 0.02, 95% confidence interval −0.34 to 0.30). Rater leniency is of large magnitude when making daily clinical evaluations, even when using a valid and psychometrically reliable instrument. Rater leniency was context dependent, not solely a reflection of raters’ personality or rating style.Keywords
This publication has 27 references indexed in Scilit:
- Reliability of ranking anesthesiologists and nurse anesthetists using leniency-adjusted clinical supervision and work habits scoresJournal of Clinical Anesthesia, 2020
- Hypotension during induction of anaesthesia is neither a reliable nor a useful quality measure for comparison of anaesthetists’ performanceBritish Journal of Anaesthesia, 2017
- Operating Room Anesthesia Subspecialization Is Not Associated With Significantly Greater Quality of Supervision of Anesthesia Residents and Nurse AnesthetistsAnesthesia & Analgesia, 2017
- Measurement of faculty anesthesiologists’ quality of clinical supervision has greater reliability when controlling for the leniency of the rating anesthesia resident: a retrospective cohort studyCanadian Journal of Anesthesia/Journal canadien d'anesthésie, 2017
- Do Not Use Hierarchical Logistic Regression Models with Low-incidence Outcome Data to Compare Anesthesiologists in Your DepartmentAnesthesiology, 2016
- Work Habits Are Valid Components of Evaluations of Anesthesia Residents Based on Faculty Anesthesiologists’ Daily Written Comments About ResidentsAnesthesia & Analgesia, 2016
- Prolonged Operative Time to Extubation Is Not a Useful Metric for Comparing the Performance of Individual Anesthesia ProvidersAnesthesiology, 2016
- Assessing and Comparing Anesthesiologists’ Performance on Mandated Metrics Using a Bayesian ApproachAnesthesiology, 2015
- Bernoulli Cumulative Sum (CUSUM) Control Charts for Monitoring of Anesthesiologists’ Performance in Supervising Anesthesia Residents and Nurse AnesthetistsAnesthesia & Analgesia, 2014
- Influence of Provider Type (Nurse Anesthetist or Resident Physician), Staff Assignments, and Other Covariates on Daily Evaluations of Anesthesiologists’ Quality of SupervisionAnesthesia & Analgesia, 2014