Kent G. Hecker

A blueprint is a template for an assessment that outlines the knowledge, skills, attributes, species, and systems to be assessed and the respective weightings (percentage of the assessment) for the key content areas. This is a foundational step in building content validity for an assessment method. The rapid blueprinting process is reported to be a quick and cost-effective approach to developing a blueprint that consists of two phases. The first is an in-person advisory committee meeting consisting of educational experts and key stakeholders (10–15 people) to build the framework of the assessment, based on identified foundational knowledge and skills (common across schools) at a specific time point (e.g., graduation). Key documents used to guide these discussions could include information on the type of assessment (low-stakes or high-stakes assessments), the types of items to be used (i.e., multiple-choice questions), and competency frameworks from which to build the assessment (i.e., the Competency-Based Veterinary Education framework and the Council on Education competencies). The second phase involves engaging the broader community in a validation step, via a modified Delphi procedure, to review the content and provide emphasis on those areas. Here, we outline the steps of the rapid blueprinting process to build a test blueprint for academic veterinary medicine progress test in North America. This method could assist both schools and post-graduate veterinary specialty colleges in building test or curriculum blueprints in a rapid, cost-effective manner and to develop validity evidence for the use and interpretation of test scores.

Introduction: Veterinary admissions committees aim to select applicants who will successfully complete a Doctor of Veterinary Medicine (DVM) program and contribute effectively to the profession. Although traditional admission procedures emphasize academic indicators such as undergraduate grade point average (uGPA) and standardized test scores, increasing attention has been placed on assessing social–emotional skills and school-specific contextual attributes. Given the high-stakes nature of admissions and the claim that multiple attributes are assessed at the time of selection, internal structure validity evidence is required to support the use of scores from admissions measures in the selection process. This study examines whether commonly used admissions measures reflect distinct academic, social–emotional, and contextual factors. Methods: Participants included 338 applicants (78.17% female) in a retrospective cohort study from the 2022–2024 admissions cycles at the Faculty of Veterinary Medicine, University of Calgary, Canada. Admissions measures included uGPA, Medical College Admissions Test (MCAT) total and subscores, Multiple Mini-Interview (MMI) scores, Computer-based Assessment for Sampling Personal Characteristics (Casper) situational judgement scores, standardized essay ratings, and structured file review scores. Bivariate correlation and Exploratory Factor Analysis (EFA) were used to explore the relationships and internal structure between the scores from these admissions measures. Results: Correlations among admissions variables showed small to moderate associations, with expected clustering among academic indicators. EFA produced a three-factor solution accounting for 53.36% of the variance which were labeled: (1) Academic Ability (uGPA; MCAT academic subscores), (2) social– emotional skills (essay, Casper, MCAT social emotional sub score; partial MMI loading), and (3) contextual factors (file review; partial MMI loading). Discussion: Findings provide internal structure validity evidence supporting the claim that admissions measures capture multiple attributes and these measures should be considered collectively as a system of assessment. Notably, the MMI contributed to both social–emotional and contextual factors, highlighting its multifaceted nature. These findings provide a foundation for longitudinal research examining the predictive validity of academic, social–emotional, and contextual factors for performance throughout veterinary training.

Performance-based assessments (PBAs) are essential tools in veterinary medical education for evaluating students’ clinical and professional competencies. This reflection on assessment is grounded in the first author’s experience as a veterinary educator navigating the complexities of PBAs and in the second author’s experience as a psychometrician studying validity claims about raters’ cognitive processes as they use PBAs. By integrating personal insights with existing literature, this work contributes to the ongoing discourse on assessment design and defensibility in health professional education. Our aim is to highlight the need to explore how rater response processes influence assessment outcomes and to advocate for more robust, contextually relevant validity evidence in veterinary medical education. While multiple sources of validity evidence are needed to support decisions based on PBA scores, the response processes of raters—how they interpret and apply assessment criteria—remain underexplored in health profession education. Without this type of evidence, the defensibility of assessment scores, as well as the decisions based on them, may be compromised. We argue that understanding rater decision making is crucial for strengthening the validity argument of PBAs.

Purpose: In this paper, the authors use their collective experience as medical education scholars and change agents to engage in a dialogical approach examining five myths regarding the role of contestation in curricular change. In doing so, they argue that what is taught, how it is taught and what knowledge is valued in curricula is not a neutral decision; rather, it is the result of contestation, negotiation and compromise. Approach: Five myths are contested: that curriculum planning is uncontested; that contestation is between two individuals; that curriculum is and should be the primary focus of contestation; that change is both necessary and desirable; and that consensus (or compromise) is valued over contestation. To challenge the myths, the author team engaged in a dialogic analysis prompted by a literature review on curriculum change and drawing on the authors' experiences as medical educators, which resulted in a suite of heuristics regarding how curriculum change can be both supported and opposed. Findings: Curriculum, whether it changes or not, is a site of ideological struggle. Participants should anticipate and strategically navigate disagreement. Compromise, rather than victory, is often the outcome. There is a need to go beyond the basics of what is taught, how much it is taught and when. Change should not always be the goal, not least because it can lead to superficial reforms, unintended consequences and a neglect of rigorous pedagogical evaluation. The pursuit of consensus and buy‐in from all members can inadvertently (and sometimes deliberately) stifle diverse perspectives, leading to long‐term resentment and disengagement. Implications: Educators need to identify, understand and constructively engage with different forms of contestation in curriculum planning and management. By attending to processes for engaging in contestation, it is hoped that more educators will feel empowered to raise their concerns and negotiate with others in a professional manner.

Admission to veterinary school is generally based on academic and non-academic measures. Descriptions of animal or veterinary experience and rural versus urban background are often sought from applicants, but little is objectively known about their impact on admission success or future performance. We evaluated scores from written descriptions from 590 veterinary school applicants for the nature and extent of self-reported animal experience. For those admitted to the program, we compared animal experience and rural versus urban background to performance in discipline-based courses, professional skills courses, clinical rotations, and the North American Veterinary Licensing Exam (NAVLE). More than 98% of applicants reported animal experience, with small animal veterinary experience most reported. There was no difference in animal experience or background between successful and unsuccessful applicants, but rural and urban applicants reported different experiences. There was a small correlation between small animal experience and performance in clinical rotations (.21), a small negative correlation between rural background and NAVLE performance (−.23), but otherwise, no significant correlations between animal experience or background and future performance. These findings suggest that scores of self-reported animal experience do not provide predictive information on applicants, or, alternatively, that the nature and extent of animal experience, the methods used to score these experiences, and/or the measures assessed during veterinary school need to be explicitly defined to ensure that we are capturing the appropriate information. More investigation into the scoring and impact of animal experience and background on applicant performance in the DVM program and success in a veterinary career is warranted.

Objective Communication skills assessment (CSA) is essential for ensuring competency, guiding educational practices and safeguarding regulatory compliance in health professions education (HPE). However, there appears to be heterogeneity in the reporting of validity evidence from CSA methods across the health profession that complicates our interpretation of the quality of assessment methods. Our objective was to map reliability and validity evidence from scores of CSA methods that have been reported in HPE. Design Scoping review. Data sources MEDLINE, Embase, PsycINFO, CINAHL, ERIC, CAB Abstracts and Scopus databases were searched up to March 2024. Eligibility criteria We included studies, available in English, that reported validity evidence (content-related, internal structure, relationship with other variables, response processes and consequences) for CSA methods in HPE. There were no restrictions related to date of publication. Data extraction and synthesis Two independent reviewers completed data extraction and assessed study quality using the Medical Education Research Study Quality Instrument. Data were reported using descriptive analysis (mean, median, range). Results A total of 146 eligible studies were identified, including 98 394 participants. Most studies were conducted in human medicine (124 studies) and participants were mostly undergraduate students (85 studies). Performance-based, simulated, inperson CSA was most prevalent, comprising 115 studies, of which 68 studies were objective structured clinical examination-based. Other types of methods that were reported were workplace-based assessment; asynchronous, video-based assessment; knowledge-based assessment and performance-based, simulated, virtual assessment. Included studies used a diverse range of communications skills frameworks, rating scales and raters. Internal structure was the most reported source of validity evidence (130 studies (90%), followed by content-related (108 studies (74%), relationships with other variables (86 studies (59%), response processes (15 studies (10%) and consequences (16 studies (11%). Conclusions This scoping review identified gaps in the sources of validity evidence related to assessment method that have been used to support the use of CSA methods. These gaps could be addressed by studies explicitly defining the communication skill construct(s) assessed, clarifying the validity source(s) reported and defining the intended purpose and use of the scores (ie, for learning and feedback, for decision making purposes). Our review provides a map where targeted CSA development and support are needed. Limitations of the evidence come from score interpretation being constrained by the heterogeneity of the definition of communication skills across the health professions and the reporting quality of the studies.

Veterinarians undergo several years of rigorous education in order to qualify in their chosen profession. As they enter clinical practice, or work within other areas of the profession, they embark upon a career-long journey of learning, whether that be ‘formal’ or ‘informal’ education and training, in order to develop themselves professionally and remain up to date. However, the vast majority of published educational literature within the veterinary sector relates to undergraduate programs. Research and scholarship relating to veterinary education and training beyond graduation is extremely sparse in comparison. This is somewhat different to what is seen in other health professions, including medical education, where a significant proportion of the literature focuses on education and training beyond graduation, from early career training and residencies through to continuing education. The advantages of publishing high-quality scholarship and research in any field are well known. Sharing more evidence and best practice in post-graduation education and training will inform international advances in this area. Although the specific educational challenges facing the profession at different career stages are distinct, evidence-informed approaches to educational interventions—whether that be supporting graduates’ transition into the workplace, specialty training, or continuing education—have the potential to have a positive impact on many levels, from improved patient outcomes and client satisfaction, to enhancing veterinarians’ job satisfaction and retention in the workplace. This article discusses the gaps in evidence in veterinary education and training beyond graduation, identifying some of the current challenges that could be addressed through a greater focus in this area, and their importance. In relation to graduate transition into the workplace, further work is needed to understand the optimal design and effectiveness of support programs, including coaching and mentoring for graduates. For formal post-graduate education leading to a more advanced level of practice, there is a need to better understand which approaches to teaching and assessment promote high-quality, consistent, learning experiences and outcomes. Further evidence regarding how continuing education is identified and undertaken by learners, and the corresponding impact on practice, would be valuable, and a greater understanding into feasible yet robust licensure assessments and mechanisms for revalidation are needed.

Medical mistakes made while in a state of cognitive fatigue result in diagnostic errors, psychological distress, poor patient outcomes, and potentially, loss of life. To date, cognitive fatigue in health professionals is assessed via self-report; however, the reliability and validity of these measures are often challenged. Here we propose the use of mobile electroencephalography (mEEG) in a medical context as an objective cognitive fatigue assessment that is cost effective, reliable, and efficient. We had medical students complete a simulated night on call during which they diagnosed a series of simulated medical cases. Before and after their shift, we assessed cognitive fatigue using both behavioural and mEEG measures. Our results demonstrate a decrease in the amplitude of a neural response sensitive to cognitive fatigue from the start to the end of the simulated night on call — a finding associated with an increase in cognitive fatigue. Interestingly, we observed no reltionship between the neural response and subjective self-reported cognitive fatigue scores, supporting the need for objective as opposed to subjective measures when asasessing cognitive fatigue.

Background/Objectives: When attempting to study neurocognitive mechanisms with electroencephalography (EEG) in applied ecologically valid settings, responses to stimuli may differ in time, which presents challenges to traditional EEG averaging methods. In this proof-of-concept paper, we present a method to normalize time over unequal trial lengths while preserving frequency content. Methods: Epochs are converted to time-frequency space where they are resampled to contain an equal number of timepoints representing the proportion of trial complete rather than true time. To validate this method, we used EEG data recorded from 8 novices and 4 experts in veterinary medicine while completing decision-making tasks using two question types: multiple-choice and script concordance questions used in veterinary school exams. Results: The resulting resampled time-frequency data were analyzed with partial least squares (PLS), a multivariate technique that extracts patterns of data that support a contrast between conditions and groups while controlling for Type I error. We found a significant latent variable representing a difference between question types for experts only. Conclusions: Despite within and between subject differences in timing, we found consistent differences between question types in experts in gamma and beta bands that are consistent with changes resulting from increased information load and decision-making. This novel analysis method may be a viable path forward to preserve ecological validity in EEG studies.