In‐training evaluations: developing an automated screening tool to measure report quality
- 9 June 2014
- journal article
- Published by Wiley in Medical Education
- Vol. 48 (7) , 724-732
Abstract
Objectives: In‐training evaluation ( ITE ) is used to assess resident competencies in clinical settings. This assessment is documented on an evaluation report (In‐Training Evaluation Report [ ITER ]). Unfortunately, the quality of these reports can be questionable. Therefore, training programmes to improve report quality are common. The Completed Clinical Evaluation Report Rating ( CCERR ) was developed to assess completed report quality and has been shown to do so in a reliable manner, thus enabling the evaluation of these programmes. The CCERR is a resource‐intensive instrument, which may limit its use. The purpose of this study was to create a screening measure (Proxy‐ CCERR ) that can predict the CCERR outcome in a less resource‐intensive manner. Methods: Using multiple regression, the authors analysed a dataset of 269 ITER s to create a model that can predict the associated CCERR scores. The resulting predictive model was tested on the CCERR scores for an additional sample of 300 ITER s. Results: The quality of an ITER , as measured by the CCERR , can be predicted using a model involving only three variables ( R 2 = 0.61). The predictive variables included the total number of words in the comments, the variability of the ratings and the proportion of comment boxes completed on the form. Conclusions: It is possible to model CCERR scores in a highly predictive manner. The predictive variables can be easily extracted in an automated process. Because this model is less resource‐intensive than the CCERR , it makes it possible to provide feedback from ITER training programmes to large groups of supervisors and institutions, and even to create automated feedback systems using Proxy‐ CCERR scores.Keywords
This publication has 15 references indexed in Scilit:
- Quality evaluation reports: Can a faculty development program make a difference?Medical Teacher, 2012
- Faculty Development in Assessment: The Missing Link in Competency-Based Medical EducationAcademic Medicine, 2011
- Advancing Resident Assessment in Graduate Medical EducationJournal of Graduate Medical Education, 2009
- Effect of Rater Training on Reliability and Accuracy of Mini-CEX Scores: A Randomized, Controlled TrialJournal of General Internal Medicine, 2008
- Effects of Training in Direct Observation of Medical Residents' Clinical CompetenceAnnals of Internal Medicine, 2004
- SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance RatingsTeaching and Learning in Medicine, 2003
- Assessment of Clinical Performance: In-Training Evaluation2002
- An innovative evaluation method in an internal medicine clerkshipAcademic Medicine, 1996
- Do final grades reflect written qualitative evaluations of student performance?Teaching and Learning in Medicine, 1993
- The selection and training of examiners for clinical examinationsMedical Education, 1980