Taken Out of Context: Hazards in the Interpretation of Written Assessment Comments
Authors:
- 22 October 2019
- journal article
- research article
- Published by Oxford University Press (OUP) in Academic Medicine
- Vol. 95 (7) , 1082-1088
Abstract
Purpose: Written comments are increasingly valued for assessment; however, a culture of politeness and the conflation of assessment with feedback lead to ambiguity. Interpretation requires reading between the lines, which is untenable with large volumes of qualitative data. For computer analytics to help with interpreting comments, the factors influencing interpretation must be understood. Method: Using constructivist grounded theory, the authors interviewed 17 experienced internal medicine faculty at 4 institutions between March and July, 2017, asking them to interpret and comment on 2 sets of words: those that might be viewed as “red flags” (e.g., good, improving) and those that might be viewed as signaling feedback (e.g., should, try). Analysis focused on how participants ascribed meaning to words. Results: Participants struggled to attach meaning to words presented acontextually. Four aspects of context were deemed necessary for interpretation: (1) the writer; (2) the intended and potential audiences; (3) the intended purpose(s) for the comments, including assessment, feedback, and the creation of a permanent record; and (4) the culture, including norms around assessment language. These contextual factors are not always apparent; readers must balance the inevitable need to interpret others’ language with the potential hazards of second-guessing intent. Conclusions: Comments are written for a variety of intended purposes and audiences, sometimes simultaneously; this reality creates dilemmas for faculty attempting to interpret these comments, with or without computer assistance. Attention to context is essential to reduce interpretive uncertainty and ensure that written comments can achieve their potential to enhance both assessment and feedback.Keywords
This publication has 21 references indexed in Scilit:
- Reading between the lines: faculty interpretations of narrative evaluation commentsMedical Education, 2015
- The Quality of Written Feedback by Attendings of Internal Medicine ResidentsJournal of General Internal Medicine, 2015
- How Good Is Good? Students and Assessors’ Perceptions of Qualitative Markers of PerformanceTeaching and Learning in Medicine, 2013
- Quality evaluation reports: Can a faculty development program make a difference?Medical Teacher, 2012
- Determining Need for Remediation Through Postrotation EvaluationsJournal of Graduate Medical Education, 2012
- Competencies “Plus”: The Nature of Written Comments on Internal Medicine Residentsʼ Evaluation FormsAcademic Medicine, 2011
- Programmatic assessment: From assessment of learning to assessment for learningMedical Teacher, 2011
- The “Good” Deanʼs LetterAcademic Medicine, 2010
- An Exploration of Faculty Perspectives on the In-Training Evaluation of ResidentsAcademic Medicine, 2010
- The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory.Psychological Bulletin, 1996