Benjamin Kinnear

Assessments serve several purposes in competency-based medical education (CBME) and competency-based health professions education (CBHPE) programs. These purposes are commonly termed ‘assessment for learning’ and ‘assessment of learning.’ Historically, educational scholars have advised against using an assessment for multiple purposes, arguing that doing so undermines its primary purpose. However, at times, such an approach may be required from a pragmatic standpoint. In this Eye Opener, the authors consider the competing purposes of assessment in CBHPE programs from a practical lens. They propose specific strategies to maximize benefits and minimize risks of dual-purposing when such an approach is necessary. First, the authors describe the importance of setting the stage for dual-purposing by creating a culture of trust, which requires clear institutional commitment to learner orientation and faculty development, infrastructure, and appropriate channels of communication with all constituents. Second, data considerations are discussed. The authors focus on adoption of programmatic assessment principles, opportunities for frequent formative observations, and strategies for leveraging available technology. Finally, the focus turns to the learner, highlighting their essential function as a key agent in driving their own growth and development. Balancing learner- and supervisor-initiated assessments, implementing longitudinal coaching programs, and considering “ultra-low-stakes” assessments are identified as mechanisms to support learner agency and development. Taken together, these specific design conditions may be used to promote learner growth and acquisition of competence when practical constraints require assessment data to serve a dual purpose.

Assessment in health professions education has evolved dramatically in the past hundred years. The way in which we conceptualize validity has also required evolution, particularly in the era of competency-based health professions education (CBHPE). The CBHPE approach compels us to reexamine a historical focus on objectivity, individual assessments, and standards of achievement alone. In this manuscript, the authors explore the implications CBHPE places on validity, highlighting three tension points: the paradigm of assessment, levels of validation, and the reference for competence. The authors examine health profession education’s historical orientation toward the post-positivist paradigm including an emphasis on standardization, reliability, and truth-seeking. This is contrasted with two alternative paradigms, constructivism and pragmatism, which prioritize different data sources to build validity arguments. In the section on levels of validation, the authors explore tensions between validity arguments that are holistic (considers the entire picture of performance) and those which use a decomposition approach (distilling competence into individual components). Finally, the authors consider the value in measuring learner competence using norm-referenced standards, criterion-reference approaches, or the measurement of growth throughout (and beyond) their training. Ultimately, the authors propose a balanced approach to address each tension, considering the strengths and weaknesses of traditional methods with the modern needs of CBHPE.

Introduction: Validity is a term that carries significant weight within health professions education (HPE), particularly with regard to learner assessment. Validity theory has changed significantly over the last century, transforming in concert with education, measurement and assessment practices. Most recently, HPE has adopted the conceptualisation of validity as an argument supporting the interpretation (and sometimes uses) of assessment data. The spread of competency‐based education and programmatic assessment has introduced new tensions and opportunities for operationalising argument‐based approaches to validity. Methods: The authors used a selective interpretive synthesis of the literature to explore how current argument‐based validity frameworks can be extended to support the complexity of programmatic assessment and to meet the social imperative of public protection that is core to competency‐based education. Results: The authors advocate that argument‐based frameworks, such as those of Messick or Kane, are useful in applying validity theory to competency‐based education, particularly when navigating the complexity of programmatic assessment and the increasing emphasis on consequences of assessment decisions. The authors propose three ways that can help with validity and validation in competency‐based systems: argumentation theory, integration of qualitative principles in validation and evidence‐centred design. Discussion: Argument‐based validity conceptualisations provide flexibility to accommodate competency‐based assessment systems and allow for the foregrounding of consequences of assessment decisions during validation.

Background: Medical education has moved beyond debating whether objectivity in assessment is achievable, acknowledging that assessment is inherently subjective. Yet the pursuit of objectivity, through standardisation, bias reduction and statistical rigour, remains a cherished goal. This paper argues that this lingering pursuit of objectivity undermines strong validity arguments and produces assessments that fail to engage meaningfully with the lived realities and possible futures of marginalised learners and patients. Theoretical Framework: Drawing on justice‐oriented assessment validity (JAV) frameworks and disability critical race theory (DisCrit), we examine four pervasive myths in medical education assessment: (1) Constructs are universal and stable, (2) removing bias from assessment is sufficient, (3) flexibility decreases rigour and (4) numbers are neutral. These myths allow educators to make validity arguments that ignore or explain away harm. We demonstrate how each myth, when examined from the standpoint of disabled, racially and ethnically marginalised learners, reveals itself as both inaccurate and harmful. Conclusions: We argue that validity arguments are not merely technical exercises but moral endeavours that determine whose knowledge counts and who is centred or excluded in defining competence. Rooted in the eugenic origins of educational measurement, contemporary assessment practices continue to privilege dominant group norms while systematically disadvantaging marginalised individuals and groups. A justice‐oriented approach, informed by JAV and DisCrit, requires centring marginalised perspectives, embracing flexibility as essential for accurate measurement, critically examining how numbers can obscure inequities and demonstrating that assessments work for those most likely to be harmed by them.

There is growing interest in coaching as a strategy to support competency‐based medical education (CBME), individualized learning, development of Master Adaptive Learners, and precision education. Despite high enthusiasm, educators often struggle with how to begin developing an effective program. This educational blueprint describes two foundational decisions for launching a coaching program: defining program goals and selecting a theory of change with an aligned coaching model. We then examine contextual considerations, implementation challenges, and tradeoffs that influence these decisions. Grounding a program in explicit goals, educational theory and intentional model selection create a coherent foundation for subsequent design, implementation, and evaluation.

Purpose: To develop TRainee Attributable & Automatable Care Evaluations in Real‑time (TRACERs) for inpatient diabetes management, collect validity evidence for their use in formative assessment, and explore performance variation across residents and institutions. Method: In 2023, a multi‑institutional team created 2 TRACERs based on type 2 diabetes guidelines—discourage bolus‑only insulin (TRACER #1) and encourage basal (± bolus) insulin (TRACER #2)—for internal medicine residents at 3 large residency programs. Residents were attributed to inpatient admissions based on placing the most medication orders in the first 12 hours. Structured queries extracted 35 discrete variables from the electronic health record. Two experts per institution reviewed random admissions (July-August 2022) to establish criterion validity. A retrospective cohort (July 2020-June 2023) added validity evidence. Results: Automated extraction achieved ≥96% sensitivity and ≥95% specificity when compared to manual review. Among 615 residents attributed to 6192 admissions of patients with type 2 diabetes at high risk for hyperglycemia, TRACER #1 occurred in 42.6% (1689/3965) of admissions at Program A, 28.9% (408/1410) at Program B, and 26.7% (218/817) at Program C. TRACER #2 occurred in 44.9% (367/817) of Program C admissions versus 24.2% (959/3965) at Program A and 28.2% (397/1410) at Program B (all P < .001). Four resident-level insulin‑ordering profiles were identified—consistent basal-bolus insulin use (most guideline-concordant), basal-predominant, bolus-predominant, and bolus-only (most guideline-discordant)—with between‑resident variation exceeding between‑program differences. Longitudinally, cohort-level trends masked opposing individual trajectories—some residents improved with exposure while others worsened—and performance tertiles were distinguishable early in training. Conclusions: TRACERs revealed substantial resident‑ and institution‑level variation in insulin‑ordering practices, including guideline deviations, demonstrating potential for real‑time formative feedback. Multi‑institutional implementation highlighted scalability barriers, including electronic health record heterogeneity and workflow‑dependent attribution, underscoring the need for continued refinement and broader validation.

Introduction: Inequitable assessments in medical education disproportionately affect historically marginalized trainees and undermine development of a diverse physician workforce. With programs around the world incorporating competency-based medical education (CBME), understanding whether assessment disparities exist in specialty-specific competency frameworks is critical. This study examined disparities in program-reported performance assessments of pediatric residents across demographic groups in two pediatric-specific competency frameworks: the General Pediatrics Entrustable Professional Activities (EPAs) and Accreditation Council for Graduate Medical Education (ACGME) pediatric milestones. Methods: This prospective cohort study was conducted at 15 pediatric residency programs in the U.S. across the 2021–22, 2022–23, and 2023–24 academic years. Clinical Competency Committee-assigned entrustment-supervision levels for 5 EPAs and all milestone levels were collected twice yearly. Residents self-reported ethnoracial group and sex, analyzed as exposure to systems of racism and sexism. Time-to-event analysis and growth curve modeling with intersectional analytic approaches examined disparities across demographic groups. Results: 475 of 803 (59%) residents completed demographic surveys, including 17.3% underrepresented in medicine (URiM), 22.9% Asian, 56.4% white, 70.7% female, and 27.8% male. Time-to-event analyses revealed significant differences for Asian and Asian female residents in reaching the highest EPA supervision level across all EPAs. Growth curve analyses showed Asian males had significantly flatter growth trajectories for EPA 15 (Lead an Interprofessional Team), resulting in lower estimated scores at graduation. URiM female residents had significantly lower scores on EPA 16 (Facilitate Handovers) at graduation. URiM residents showed lower Milestone growth curves for interpersonal communication, and Asian residents for systems-based practice. Discussion: Subtle but meaningful disparities exist in program-reported assessments of pediatric residents, with distinct patterns affecting Asian and URiM residents across both frameworks. These findings highlight the need for proactive attention to equity in competency-based medical education assessment systems.