Consequences Validity Evidence: Evaluating the Impact of Educational Assessments
- 1 June 2016
- journal article
- Published by Oxford University Press (OUP) in Academic Medicine
- Vol. 91 (6) , 785-795
Abstract
Because tests that do not alter management (i.e., influence decisions and actions) should not be performed, data on the consequences of assessment constitute a critical source of validity evidence. Consequences validity evidence is challenging for many educators to understand, perhaps because it has no counterpart in the older framework of content, criterion, and construct validity. The authors’ purpose is to explain consequences validity evidence and propose a framework for organizing its collection and interpretation. Both clinical and educational assessments can be viewed as interventions. The act of administering or taking a test, the interpretation of scores, and the ensuing decisions and actions influence those being assessed (e.g., patients or students) and other people and systems (e.g., physicians, teachers, hospitals, schools). Consequences validity evidence examines such impacts of assessments. Despite its importance, consequences evidence is reported infrequently in health professions education (range 5%–20% of studies in recent systematic reviews) and is typically limited in scope and rigor. Consequences validity evidence can derive from evaluations of the impact on examinees, educators, schools, or the end target of practice (e.g., patients or health care systems); and the downstream impact of classifications (e.g., different score cut points and labels). Impact can result from the uses of scores or from the assessment activity itself, and can be intended or unintended and beneficial or harmful. Both quantitative and qualitative research methods are useful. The type, quantity, and rigor of consequences evidence required will vary depending on the assessment and the claims for its use.Keywords
This publication has 45 references indexed in Scilit:
- Colorectal-Cancer Incidence and Mortality with Screening Flexible SigmoidoscopyNew England Journal of Medicine, 2012
- Procedural Competence in Internal Medicine Residents: Validity of a Central Venous Catheter Insertion Assessment InstrumentAcademic Medicine, 2009
- Do Metrics Matter? Time Versus Motion Tracking for Performance Assessment of Proficiency-Based Laparoscopic Skills TrainingSimulation in Healthcare: The Journal of the Society for Simulation in Healthcare, 2009
- Strengths and weaknesses in the consultation skills of senior medical students: identification, enhancement and curricular changeMedical Education, 2006
- Evaluation of the airway of the SimMan™ full‐scale patient simulatorActa Anaesthesiologica Scandinavica, 2005
- A Clinical-performance Biopsy InstrumentAcademic Medicine, 2002
- Strategies for Examining the Consequences of Assessment and Accountability ProgramsEducational Measurement: Issues and Practice, 2002
- Consequential Validity From the Test Developer's PerspectiveEducational Measurement: Issues and Practice, 1998
- A Model For Evaluating the History Taking and Physical Examination Skills of Medical StudentsMedical Teacher, 1989
- Evaluating a direct observation exercise in a psychiatric clerkshipMedical Education, 1984