Validity Evidence for a Patient Note Scoring Rubric Based on the New Patient Note Format of the United States Medical Licensing Examination
Open Access
- 1 October 2013
- journal article
- rime
- Published by Oxford University Press (OUP) in Academic Medicine
- Vol. 88 (10) , 1552-1557
Abstract
Purpose This study examines validity evidence for the Patient Note Scoring Rubric, which was developed for a local graduation competency exam (GCE) to assess patient notes written in the new United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills format. The rubric was designed to measure three dimensions: Documentation, justified differential diagnosis (DDX), and Workup. Method Analyses used GCE data from 170 fourth-year medical students who completed five standardized patient (SP) cases in May 2012. Five physician raters each scored all responses for one case. Internal structure was examined using correlations between dimensions and between cases; a generalizability study was also conducted. Relationship to other variables was examined by correlating patient note scores with SP encounter scores. Consequence was assessed by comparing pass–fail rates between the rubric and the previous global rating. Response process was examined using rater feedback. Results Correlations between scores from different dimensions ranged between 0.33 and 0.44. Reliability of scores based on the phi coefficient was 0.43; 15 cases were required to reach a phi coefficient of 0.70. Evidence of case specificity was found. Documentation scores were moderately correlated with SP scores for data gathering (r = 0.47, P < .001). There was no meaningful change in pass–fail rates. Raters’ feedback indicated that they required more training for scoring the DDX and Workup dimensions. Conclusions There is initial validity evidence for use of this rubric to score local clinical exams that are based on the new USMLE patient note format.Keywords
This publication has 8 references indexed in Scilit:
- Biomedical knowledge, clinical cognition and diagnostic justification: a structural equation modelMedical Education, 2013
- Rater Cognition: Implications for ValidityEducational Measurement: Issues and Practice, 2012
- Examining the Diagnostic Justification Abilities of Fourth-Year Medical StudentsAcademic Medicine, 2012
- A hypothesis‐driven physical examination learning and assessment procedure for medical students: initial validity evidenceMedical Education, 2009
- An Examination of Rater Drift Within a Generalizability Theory FrameworkJournal of Educational Measurement, 2009
- Changes in clinical skills education resulting from the introduction of the USMLE™ step 2 clinical skills (CS) examinationMedical Teacher, 2008
- Quality control of an OSCE using generalizability theory and many-faceted Rasch measurementAdvances in Health Sciences Education, 2007
- Standards of Validity and the Validity of Standards in Performance AsessmentEducational Measurement: Issues and Practice, 1995