Jennifer R. Kogan
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Jennifer R. Kogan has a total of 36 co-authors. Here are the top co-authors by number of shared publications:
- Eric S. Holmboe
Accreditation Council for Graduate Medical Education
5 shared publications - C. Jessica Dine
Perelman School of Medicine, University of Pennsylvania, Philadelphia, USA
3 shared publications - Lisa N. Conforti
Lisa N. Conforti, MPH, is Senior Research Analyst, Accreditation Council for Graduate Medical Education (ACGME), Chicago, Illinois, USA
3 shared publications - Arnyce R. Pock
Uniformed Services, University of the Health Sciences, Bethesda, Maryland, USA
2 shared publications - Aubrie Swan Sein
Vagelos College of Physicians and Surgeons Columbia University New York New York USA
2 shared publications - Christine Seibert
Medicine, University of Wisconsin School of Medicine and Public Health United States
2 shared publications
Latest Publications
- Open Access
- 24 October 2026
- Published by Oxford University Press (OUP) in Academic Medicine
Comparative analysis of large language model and faculty rater scoring of medical student documentation in objective structured clinical examinations
Purpose: To evaluate whether a commercially available large language model (LLM) can score medical student post-encounter notes from Objective Structured Clinical Examinations (OSCEs) using an existing institutional rubric without task-specific model training and to characterize internal consistency and alignment with faculty scores using absolute agreement and rank-order association metrics. Method: The dataset included 433 medical student notes from 7 standardized OSCE cases at the University of Kentucky. In November 2024, an LLM (OpenAI GPT-4o mini, via API) scored each note using a prompt-based workflow (no model fine-tuning) under two approaches: numerical score only (“GPT-NE”, 30 iterations per note) and numerical score with rationale (“GPT-E”, 4 iterations per note). The authors compared the LLM’s performance against historical faculty grades using the same rubric, evaluating the reliability of grading methodologies and inter-modal agreement. Results: The LLM demonstrated high internal consistency for GPT-NE (intraclass correlation (ICC) = 0.94) and moderate-to-high consistency for GPT-E (ICC = 0.76). LLM scores were significantly lower than faculty scores (mean differences on a 1–4 scale: -1.12 GPT-NE, 95% CI -1.17 to -1.08; -0.65 GPT-E, 95% CI -0.69 to -0.61, both P < .001). Spearman rank-order correlations (r = 0.43–0.53) indicate moderate preservation of relative student ranking despite differences in absolute scores. Conclusions: The LLM demonstrated significant potential for automating OSCE note grading given its internal consistency and similar ranking of learners compared with faculty raters. Automation of grading could yield substantial time savings; however, discrepancies with absolute faculty scores underscore the need for further refinement of rubric alignment and grading methodologies.
- 24 October 2026
- Published by Oxford University Press (OUP) in Academic Medicine
Open AccessComparative analysis of large language model and faculty rater scoring of medical student documentation in objective structured clinical examinations
Purpose: To evaluate whether a commercially available large language model (LLM) can score medical student post-encounter notes from Objective Structured Clinical Examinations (OSCEs) using an existing institutional rubric without task-specific model training and to characterize internal consistency and alignment with faculty scores using absolute agreement and rank-order association metrics. Method: The dataset included 433 medical student notes from 7 standardized OSCE cases at the University of Kentucky. In November 2024, an LLM (OpenAI GPT-4o mini, via API) scored each note using a prompt-based workflow (no model fine-tuning) under two approaches: numerical score only (“GPT-NE”, 30 iterations per note) and numerical score with rationale (“GPT-E”, 4 iterations per note). The authors compared the LLM’s performance against historical faculty grades using the same rubric, evaluating the reliability of grading methodologies and inter-modal agreement. Results: The LLM demonstrated high internal consistency for GPT-NE (intraclass correlation (ICC) = 0.94) and moderate-to-high consistency for GPT-E (ICC = 0.76). LLM scores were significantly lower than faculty scores (mean differences on a 1–4 scale: -1.12 GPT-NE, 95% CI -1.17 to -1.08; -0.65 GPT-E, 95% CI -0.69 to -0.61, both P < .001). Spearman rank-order correlations (r = 0.43–0.53) indicate moderate preservation of relative student ranking despite differences in absolute scores. Conclusions: The LLM demonstrated significant potential for automating OSCE note grading given its internal consistency and similar ranking of learners compared with faculty raters. Automation of grading could yield substantial time savings; however, discrepancies with absolute faculty scores underscore the need for further refinement of rubric alignment and grading methodologies.
- Open Access
- 30 April 2026
- Published by Taylor & Francis in Teaching and Learning in Medicine
- p. 1-11
Does the Timing of USMLE Step 1 Affect Performance? A Comparison of Pre- and Post-Clerkship Timing in the Pass/Fail Era
The United States Medical Licensing Examination (USMLE) Step 1 has long served as a high-stakes assessment of foundational science knowledge for the practice of medicine (United States Medical Lice...
- 30 April 2026
- Published by Taylor & Francis in Teaching and Learning in Medicine
- p. 1-11
Open AccessDoes the Timing of USMLE Step 1 Affect Performance? A Comparison of Pre- and Post-Clerkship Timing in the Pass/Fail Era
The United States Medical Licensing Examination (USMLE) Step 1 has long served as a high-stakes assessment of foundational science knowledge for the practice of medicine (United States Medical Lice...
- 24 September 2024
- Published by Oxford University Press (OUP) in Academic Medicine
- Vol. 100 (2) , 137-143
Stepping Back: How Should Pass/Fail Scoring Influence Step 1 Timing?
Although most students complete Step 1 before clerkships, some institutions delay the exam until after clerkships. The change to pass/fail grading adds additional complexity that should be considered when deciding about exam timing. Both early and late administration may affect learning outcomes, learner behavior, student well-being, and residency match success. Step 1 completion before clerkships promotes learning outcomes (e.g., integration and mastery of foundational material), may encourage students to focus on the curriculum, and may better prepare students for clinical science exams (CSEs). However, delaying the exam ensures that students maintain foundational knowledge and may encourage clinical educators to demonstrate basic science illustrations. An early Step 1 may affect learner behavior by allowing clerkship students to focus on clinical learning. The associated National Board of Medical Examiners performance report may also be used for Step 2 and CSE preparation. However, delaying Step 1 allows greater scheduling flexibility based on developmental milestones. Administration of Step 1 before clerkships removes a significant stressor from the clinical year and decompresses the residency application period. However, a delayed Step 1 reduces the pressure on students to engage in numerous extracurricular and research activities to distinguish themselves due to the pass/fail change. An early Step 1 exam may also lead to improved CSE performance, which is often linked to clerkship honors criteria, an increasingly valuable distinction for residency match success after the change to pass/fail. In contrast, delaying Step 1 is associated with higher first-time pass rates, which may be especially important for students at risk for failure. Medical educators and students should collaboratively approach the question of Step 1 timing, considering these factors within the context of the medical school program, curricular constraints and priorities, and students’ individual needs and goals.
- 24 September 2024
- Published by Oxford University Press (OUP) in Academic Medicine
- Vol. 100 (2) , 137-143
Stepping Back: How Should Pass/Fail Scoring Influence Step 1 Timing?
Although most students complete Step 1 before clerkships, some institutions delay the exam until after clerkships. The change to pass/fail grading adds additional complexity that should be considered when deciding about exam timing. Both early and late administration may affect learning outcomes, learner behavior, student well-being, and residency match success. Step 1 completion before clerkships promotes learning outcomes (e.g., integration and mastery of foundational material), may encourage students to focus on the curriculum, and may better prepare students for clinical science exams (CSEs). However, delaying the exam ensures that students maintain foundational knowledge and may encourage clinical educators to demonstrate basic science illustrations. An early Step 1 may affect learner behavior by allowing clerkship students to focus on clinical learning. The associated National Board of Medical Examiners performance report may also be used for Step 2 and CSE preparation. However, delaying Step 1 allows greater scheduling flexibility based on developmental milestones. Administration of Step 1 before clerkships removes a significant stressor from the clinical year and decompresses the residency application period. However, a delayed Step 1 reduces the pressure on students to engage in numerous extracurricular and research activities to distinguish themselves due to the pass/fail change. An early Step 1 exam may also lead to improved CSE performance, which is often linked to clerkship honors criteria, an increasingly valuable distinction for residency match success after the change to pass/fail. In contrast, delaying Step 1 is associated with higher first-time pass rates, which may be especially important for students at risk for failure. Medical educators and students should collaboratively approach the question of Step 1 timing, considering these factors within the context of the medical school program, curricular constraints and priorities, and students’ individual needs and goals.
- 8 March 2024
- Published by Oxford University Press (OUP) in Academic Medicine
- Vol. 99 (9) , 1032-1037
Racial and Ethnic Bias in Letters of Recommendation in Academic Medicine: A Systematic Review
Purpose: Letters of recommendations (LORs) are key components of academic medicine applications. Given that bias against students and trainees underrepresented in medicine (UIM) has been demonstrated across assessment, achievement, and advancement domains, the authors reviewed studies on LORs to assess racial, ethnic, and UIM differences in LORs. Standardized LORs (SLORs), an increasingly common form of LORs, were also assessed for racial and ethnic differences. Method: A systematic review was conducted for English-language studies that assessed racial or ethnic differences in LORs in academic medicine published from database inception to July 16, 2023. Studies evaluating SLORs underwent data abstraction to evaluate their impact on the given race or ethnicity comparison and outcome variables. Results: Twenty-three studies describing 19,012 applicants and 41,925 LORs were included. Nineteen studies (82.6%) assessed LORs for residency, 4 (17.4%) assessed LORs for fellowship, and none evaluated employment or promotion. Fifteen of 17 studies (88.2%) assessing linguistic differences reported a significant difference in a particular race or ethnicity comparison. Of the 7 studies assessing agentic language (e.g., “strong,” “confident”), 1 study found fewer agentic terms used for Black and Latinx applicants, and 1 study reported higher agency scores for Asian applicants and applicants of races other than White. There were mixed results for the use of communal and grindstone language in UIM and non-UIM comparisons. Among 6 studies, 4 (66.7%) reported that standout language (e.g., “exceptional,” “outstanding”) was less likely to be ascribed to UIM applicants. Doubt-raising language was more frequently used for UIM trainees. When SLORs and unstructured LORs were compared, fewer linguistic differences were found in SLORs. Conclusions: There is a moderate bias against UIM candidates in the domains of linguistic differences, doubt-raising language, and topics discussed in LORs, which has implications for perceptions of competence and ability in the high-stakes residency and fellowship application process.
- 8 March 2024
- Published by Oxford University Press (OUP) in Academic Medicine
- Vol. 99 (9) , 1032-1037
Racial and Ethnic Bias in Letters of Recommendation in Academic Medicine: A Systematic Review
Purpose: Letters of recommendations (LORs) are key components of academic medicine applications. Given that bias against students and trainees underrepresented in medicine (UIM) has been demonstrated across assessment, achievement, and advancement domains, the authors reviewed studies on LORs to assess racial, ethnic, and UIM differences in LORs. Standardized LORs (SLORs), an increasingly common form of LORs, were also assessed for racial and ethnic differences. Method: A systematic review was conducted for English-language studies that assessed racial or ethnic differences in LORs in academic medicine published from database inception to July 16, 2023. Studies evaluating SLORs underwent data abstraction to evaluate their impact on the given race or ethnicity comparison and outcome variables. Results: Twenty-three studies describing 19,012 applicants and 41,925 LORs were included. Nineteen studies (82.6%) assessed LORs for residency, 4 (17.4%) assessed LORs for fellowship, and none evaluated employment or promotion. Fifteen of 17 studies (88.2%) assessing linguistic differences reported a significant difference in a particular race or ethnicity comparison. Of the 7 studies assessing agentic language (e.g., “strong,” “confident”), 1 study found fewer agentic terms used for Black and Latinx applicants, and 1 study reported higher agency scores for Asian applicants and applicants of races other than White. There were mixed results for the use of communal and grindstone language in UIM and non-UIM comparisons. Among 6 studies, 4 (66.7%) reported that standout language (e.g., “exceptional,” “outstanding”) was less likely to be ascribed to UIM applicants. Doubt-raising language was more frequently used for UIM trainees. When SLORs and unstructured LORs were compared, fewer linguistic differences were found in SLORs. Conclusions: There is a moderate bias against UIM candidates in the domains of linguistic differences, doubt-raising language, and topics discussed in LORs, which has implications for perceptions of competence and ability in the high-stakes residency and fellowship application process.
- 17 February 2024
- Vol. 16 (1) , 30-36
Accuracy of Entrustment-Based Assessment: Implications for Programs and Patients.
Background Although entrustment-supervision ratings are more intuitive compared to other rating scales, it is not known whether their use accurately assesses the appropriateness of care provided by a resident. Objective To determine the frequency of incorrect entrustment ratings assigned by faculty and whether accuracy of an entrustment-supervision scale differed by resident performance when the scripted resident performance level is known. Methods Faculty participants rated standardized residents in 10 videos using a 4-point entrustment-supervision scale. We calculated the frequency of rating a resident incorrectly. We performed generalizability (G) and decision (D) studies for all 10 cases (768 ratings) and repeated the analysis using only cases with an entrustment score of 2. Results The mean score by 77 raters for all videos was 2.87 (SD=0.86) with a mean of 2.37 (SD=0.72), 3.11 (SD=0.67) and 3.78 (SD=0.43) for the scripted levels of 2, 3, and 4. Faculty ratings differed from the scripted score for 331of 768 (43%) ratings. Most errors were ratings higher than the scripted score (223, 67%). G studies estimated the variance proportions of rater and case to be 4.99% and 54.29%. D studies estimated that 3 raters would need to watch 10 cases. The variance proportion of rater was 8.5% when the analysis was restricted to level 2 entrustment, requiring 15 raters to watch 5 cases. Conclusions Participants underestimated residents' potential need for greater supervision. Overall agreement between raters and scripted scores were low.
- 17 February 2024
- Vol. 16 (1) , 30-36
Accuracy of Entrustment-Based Assessment: Implications for Programs and Patients.
Background Although entrustment-supervision ratings are more intuitive compared to other rating scales, it is not known whether their use accurately assesses the appropriateness of care provided by a resident. Objective To determine the frequency of incorrect entrustment ratings assigned by faculty and whether accuracy of an entrustment-supervision scale differed by resident performance when the scripted resident performance level is known. Methods Faculty participants rated standardized residents in 10 videos using a 4-point entrustment-supervision scale. We calculated the frequency of rating a resident incorrectly. We performed generalizability (G) and decision (D) studies for all 10 cases (768 ratings) and repeated the analysis using only cases with an entrustment score of 2. Results The mean score by 77 raters for all videos was 2.87 (SD=0.86) with a mean of 2.37 (SD=0.72), 3.11 (SD=0.67) and 3.78 (SD=0.43) for the scripted levels of 2, 3, and 4. Faculty ratings differed from the scripted score for 331of 768 (43%) ratings. Most errors were ratings higher than the scripted score (223, 67%). G studies estimated the variance proportions of rater and case to be 4.99% and 54.29%. D studies estimated that 3 raters would need to watch 10 cases. The variance proportion of rater was 8.5% when the analysis was restricted to level 2 entrustment, requiring 15 raters to watch 5 cases. Conclusions Participants underestimated residents' potential need for greater supervision. Overall agreement between raters and scripted scores were low.
- 30 January 2024
- Published by Springer Nature in Journal of General Internal Medicine
- Vol. 39 (10) , 1795-1802
Gender Differences in Work-Based Assessment Scores and Narrative Comments After Direct Observation
Background While some prior studies of work-based assessment (WBA) numeric ratings have not shown gender differences, they have been unable to account for the true performance of the resident or explore narrative differences by gender. Objective To explore gender differences in WBA ratings as well as narrative comments (when scripted performance was known). Design Secondary analysis of WBAs obtained from a randomized controlled trial of a longitudinal rater training intervention in 2018–2019. Participating faculty (n = 77) observed standardized resident–patient encounters and subsequently completed rater assessment forms (RAFs). Subjects Participating faculty in longitudinal rater training. Main Measures Gender differences in mean entrustment ratings (4-point scale) were assessed with multivariable regression (adjusted for scripted performance, rater and resident demographics, and the interaction between study arm and time period [pre- versus post-intervention]). Using pre-specified natural language processing categories (masculine, feminine, agentic, and communal words), multivariable linear regression was used to determine associations of word use in the narrative comments with resident gender, race, and skill level, faculty demographics, and interaction between the study arm and the time period (pre- versus post-intervention). Key Results Across 1527 RAFs, there were significant differences in entrustment ratings between women and men standardized residents (2.29 versus 2.54, respectively, p < 0.001) after correction for resident skill level. As compared to men, feminine terms were more common for comments of what the resident did poorly among women residents (β 0.45, CI 0.12–0.78, p 0.01). This persisted despite adjusting for the faculty’s entrustment ratings. There were no other significant linguistic differences by gender. Conclusions Contrasting prior studies, we found entrustment rating differences in a simulated WBA which persisted after adjusting for the resident’s scripted performance. There were also linguistic differences by gender after adjusting for entrustment ratings, with feminine terms being used more frequently in comments about women in some, but not all narrative comments.
- 30 January 2024
- Published by Springer Nature in Journal of General Internal Medicine
- Vol. 39 (10) , 1795-1802
Gender Differences in Work-Based Assessment Scores and Narrative Comments After Direct Observation
Background While some prior studies of work-based assessment (WBA) numeric ratings have not shown gender differences, they have been unable to account for the true performance of the resident or explore narrative differences by gender. Objective To explore gender differences in WBA ratings as well as narrative comments (when scripted performance was known). Design Secondary analysis of WBAs obtained from a randomized controlled trial of a longitudinal rater training intervention in 2018–2019. Participating faculty (n = 77) observed standardized resident–patient encounters and subsequently completed rater assessment forms (RAFs). Subjects Participating faculty in longitudinal rater training. Main Measures Gender differences in mean entrustment ratings (4-point scale) were assessed with multivariable regression (adjusted for scripted performance, rater and resident demographics, and the interaction between study arm and time period [pre- versus post-intervention]). Using pre-specified natural language processing categories (masculine, feminine, agentic, and communal words), multivariable linear regression was used to determine associations of word use in the narrative comments with resident gender, race, and skill level, faculty demographics, and interaction between the study arm and the time period (pre- versus post-intervention). Key Results Across 1527 RAFs, there were significant differences in entrustment ratings between women and men standardized residents (2.29 versus 2.54, respectively, p < 0.001) after correction for resident skill level. As compared to men, feminine terms were more common for comments of what the resident did poorly among women residents (β 0.45, CI 0.12–0.78, p 0.01). This persisted despite adjusting for the faculty’s entrustment ratings. There were no other significant linguistic differences by gender. Conclusions Contrasting prior studies, we found entrustment rating differences in a simulated WBA which persisted after adjusting for the resident’s scripted performance. There were also linguistic differences by gender after adjusting for entrustment ratings, with feminine terms being used more frequently in comments about women in some, but not all narrative comments.
- 18 April 2023
- Published by Oxford University Press (OUP) in Academic Medicine
- Vol. 98 (Supplement) , S37-S49
The Urgency of Now: Rethinking and Improving Assessment Practices in Medical Education Programs
Assessment is essential to professional development. Assessment provides the information needed to give feedback, support coaching and the creation of individualized learning plans, inform progress decisions, determine appropriate supervision levels, and, most importantly, help ensure patients and families receive high-quality, safe care in the training environment. While the introduction of competency-based medical education has catalyzed advances in assessment, much work remains to be done. First, becoming a physician (or other health professional) is primarily a developmental process, and assessment programs must be designed using a developmental and growth mindset. Second, medical education programs must have integrated programs of assessment that address the interconnected domains of implicit, explicit and structural bias. Third, improving programs of assessment will require a systems-thinking approach. In this paper, the authors first address these overarching issues as key principles that must be embraced so that training programs may optimize assessment to ensure all learners achieve desired medical education outcomes. The authors then explore specific needs in assessment and provide suggestions to improve assessment practices. This paper is by no means inclusive of all medical education assessment challenges or possible solutions. However, there is a wealth of current assessment research and practice that medical education programs can use to improve educational outcomes and help reduce the harmful effects of bias. The authors’ goal is to help improve and guide innovation in assessment by catalyzing further conversations.
- 18 April 2023
- Published by Oxford University Press (OUP) in Academic Medicine
- Vol. 98 (Supplement) , S37-S49
The Urgency of Now: Rethinking and Improving Assessment Practices in Medical Education Programs
Assessment is essential to professional development. Assessment provides the information needed to give feedback, support coaching and the creation of individualized learning plans, inform progress decisions, determine appropriate supervision levels, and, most importantly, help ensure patients and families receive high-quality, safe care in the training environment. While the introduction of competency-based medical education has catalyzed advances in assessment, much work remains to be done. First, becoming a physician (or other health professional) is primarily a developmental process, and assessment programs must be designed using a developmental and growth mindset. Second, medical education programs must have integrated programs of assessment that address the interconnected domains of implicit, explicit and structural bias. Third, improving programs of assessment will require a systems-thinking approach. In this paper, the authors first address these overarching issues as key principles that must be embraced so that training programs may optimize assessment to ensure all learners achieve desired medical education outcomes. The authors then explore specific needs in assessment and provide suggestions to improve assessment practices. This paper is by no means inclusive of all medical education assessment challenges or possible solutions. However, there is a wealth of current assessment research and practice that medical education programs can use to improve educational outcomes and help reduce the harmful effects of bias. The authors’ goal is to help improve and guide innovation in assessment by catalyzing further conversations.
- 6 January 2023
- Published by Oxford University Press (OUP) in Academic Medicine
- Vol. 98 (7) , 844-850
Gender-Based Differences in Language Used by Students to Describe Their Noteworthy Characteristics in Medical Student Performance Evaluations
Purpose: The noteworthy characteristic (NC) section of the medical student performance evaluation (MSPE) was introduced to facilitate holistic review of residency applications and mitigate biases. The student-written aspect of the characteristics, however, may introduce biases resulting from gender differences in self-promotion behaviors. The authors conducted an exploratory analysis of potential gender-based differences in language used in NCs. Method: The authors performed a single-center cohort analysis of all student-written NCs at the Perelman School of Medicine (2018–2022). NCs were converted into single words and characterized into word categories: ability (e.g., “talent”), standout (“best”), grindstone (“meticulous”), communal (“caring”), or agentic (“ambitious”). The authors qualitatively analyzed NC topic characteristics (i.e., focused on scholarship, community service). Logistic regression was used to identify gender differences in word categories and topics used in NCs. Results: The cohort included 2,084 characteristics from 783 MSPEs (47.5%, n = 371 written by women). After adjusting for underrepresented in medicine status, honor society membership, and intended specialty, men were more likely to use standout (OR = 2.00; 95% confidence interval (CI) = 1.35, 2.96; P = .001) and communal (OR = 1.40; 95% CI = 1.03, 1.90; P = .03) words in their NCs compared to women but less likely to use grindstone words (OR = 0.72; 95% CI = 0.53, 0.98; P = .04). Men were more likely than women to discuss scholarship (OR = 2.03, 95% CI = 1.27, 3.23; P = .003), hobbies (OR = 1.45; 95% CI = 1.07, 1.96; P = .02), and/or awards (OR = 1.59; 95% CI = 1.16, 2.16; P = .004) and less likely to highlight community service (OR = 0.66; 95% CI = 0.48 ,0.92; P = .02). Conclusions: The self-written nature of NCs permits language differences that may contribute to gender bias in residency application.
- 6 January 2023
- Published by Oxford University Press (OUP) in Academic Medicine
- Vol. 98 (7) , 844-850
Gender-Based Differences in Language Used by Students to Describe Their Noteworthy Characteristics in Medical Student Performance Evaluations
Purpose: The noteworthy characteristic (NC) section of the medical student performance evaluation (MSPE) was introduced to facilitate holistic review of residency applications and mitigate biases. The student-written aspect of the characteristics, however, may introduce biases resulting from gender differences in self-promotion behaviors. The authors conducted an exploratory analysis of potential gender-based differences in language used in NCs. Method: The authors performed a single-center cohort analysis of all student-written NCs at the Perelman School of Medicine (2018–2022). NCs were converted into single words and characterized into word categories: ability (e.g., “talent”), standout (“best”), grindstone (“meticulous”), communal (“caring”), or agentic (“ambitious”). The authors qualitatively analyzed NC topic characteristics (i.e., focused on scholarship, community service). Logistic regression was used to identify gender differences in word categories and topics used in NCs. Results: The cohort included 2,084 characteristics from 783 MSPEs (47.5%, n = 371 written by women). After adjusting for underrepresented in medicine status, honor society membership, and intended specialty, men were more likely to use standout (OR = 2.00; 95% confidence interval (CI) = 1.35, 2.96; P = .001) and communal (OR = 1.40; 95% CI = 1.03, 1.90; P = .03) words in their NCs compared to women but less likely to use grindstone words (OR = 0.72; 95% CI = 0.53, 0.98; P = .04). Men were more likely than women to discuss scholarship (OR = 2.03, 95% CI = 1.27, 3.23; P = .003), hobbies (OR = 1.45; 95% CI = 1.07, 1.96; P = .02), and/or awards (OR = 1.59; 95% CI = 1.16, 2.16; P = .004) and less likely to highlight community service (OR = 0.66; 95% CI = 0.48 ,0.92; P = .02). Conclusions: The self-written nature of NCs permits language differences that may contribute to gender bias in residency application.
- 21 July 2022
- Published by Oxford University Press (OUP) in Academic Medicine
- Vol. 98 (2) , 237-247
Can Rater Training Improve the Quality and Accuracy of Workplace-Based Assessment Narrative Comments and Entrustment Ratings? A Randomized Controlled Trial
Purpose: Prior research evaluating workplace-based assessment (WBA) rater training effectiveness has not measured improvement in narrative comment quality and accuracy, nor accuracy of prospective entrustment-supervision ratings. The purpose of this study was to determine whether rater training, using performance dimension and frame of reference training, could improve WBA narrative comment quality and accuracy. A secondary aim was to assess impact on entrustment rating accuracy. Method: This single blind, multi-institution, randomized controlled trial of a multifaceted, longitudinal rater training intervention consisted of in-person training followed by asynchronous online spaced learning. In 2018, investigators randomized 94 internal medicine and family medicine physicians involved with resident education. Participants assessed 10 scripted standardized resident-patient videos at baseline and follow-up. Differences in holistic assessment of narrative comment accuracy and specificity, accuracy of individual scenario observations, and entrustment rating accuracy were evaluated with t-tests. Linear regression assessed impact of participant demographics and baseline performance. Results: Seventy-seven participants completed the study. At follow-up, the intervention group (n = 41), compared to the control group (n = 36), had higher scores for narrative holistic specificity (2.76 vs 2.31, P < .001, Cohen V = .30), accuracy (2.37 vs 2.06, P < .001, Cohen V = .20) and mean quantity of accurate (6.14 vs 4.33, P < .001), inaccurate (3.53 vs 2.41, P < .001), and overall observations (2.61 vs 1.92, P = .002, Cohen V = .47). In aggregate, the intervention group had more accurate entrustment ratings (58.1% vs 49.7%, P = .006, Phi = .30). Baseline performance was significantly associated with performance on final assessments. Conclusions: Quality and specificity of narrative comments improved with rater training; the effect was mitigated by inappropriate stringency. Training improved accuracy of prospective entrustment-supervision ratings but the effect was more limited. Participants with lower baseline rating skill may benefit most from training.
- 21 July 2022
- Published by Oxford University Press (OUP) in Academic Medicine
- Vol. 98 (2) , 237-247
Can Rater Training Improve the Quality and Accuracy of Workplace-Based Assessment Narrative Comments and Entrustment Ratings? A Randomized Controlled Trial
Purpose: Prior research evaluating workplace-based assessment (WBA) rater training effectiveness has not measured improvement in narrative comment quality and accuracy, nor accuracy of prospective entrustment-supervision ratings. The purpose of this study was to determine whether rater training, using performance dimension and frame of reference training, could improve WBA narrative comment quality and accuracy. A secondary aim was to assess impact on entrustment rating accuracy. Method: This single blind, multi-institution, randomized controlled trial of a multifaceted, longitudinal rater training intervention consisted of in-person training followed by asynchronous online spaced learning. In 2018, investigators randomized 94 internal medicine and family medicine physicians involved with resident education. Participants assessed 10 scripted standardized resident-patient videos at baseline and follow-up. Differences in holistic assessment of narrative comment accuracy and specificity, accuracy of individual scenario observations, and entrustment rating accuracy were evaluated with t-tests. Linear regression assessed impact of participant demographics and baseline performance. Results: Seventy-seven participants completed the study. At follow-up, the intervention group (n = 41), compared to the control group (n = 36), had higher scores for narrative holistic specificity (2.76 vs 2.31, P < .001, Cohen V = .30), accuracy (2.37 vs 2.06, P < .001, Cohen V = .20) and mean quantity of accurate (6.14 vs 4.33, P < .001), inaccurate (3.53 vs 2.41, P < .001), and overall observations (2.61 vs 1.92, P = .002, Cohen V = .47). In aggregate, the intervention group had more accurate entrustment ratings (58.1% vs 49.7%, P = .006, Phi = .30). Baseline performance was significantly associated with performance on final assessments. Conclusions: Quality and specificity of narrative comments improved with rater training; the effect was mitigated by inappropriate stringency. Training improved accuracy of prospective entrustment-supervision ratings but the effect was more limited. Participants with lower baseline rating skill may benefit most from training.
- Open Access
- 21 July 2022
- Published by Oxford University Press (OUP) in Academic Medicine
- Vol. 97 (8) , 1128-1136
Will Any Road Get You There? Examining Warranted and Unwarranted Variation in Medical Education
Undergraduate and graduate medical education have long embraced uniqueness and variability in curricular and assessment approaches. Some of this variability is justified (warranted or necessary variation), but a substantial portion represents unwarranted variation. A primary tenet of outcomes-based medical education is ensuring that all learners acquire essential competencies to be publicly accountable to meet societal needs. Unwarranted variation in curricular and assessment practices contributes to suboptimal and variable educational outcomes and, by extension, risks graduates delivering suboptimal health care quality. Medical education can use lessons from the decades of study on unwarranted variation in health care as part of efforts to continuously improve the quality of training programs. To accomplish this, medical educators will first need to recognize the difference between warranted and unwarranted variation in both clinical care and educational practices. Addressing unwarranted variation will require cooperation and collaboration between multiple levels of the health care and educational systems using a quality improvement mindset. These efforts at improvement should acknowledge that some aspects of variability are not scientifically informed and do not support desired outcomes or societal needs. This perspective examines the correlates of unwarranted variation of clinical care in medical education and the need to address the interdependency of unwarranted variation occurring between clinical and educational practices. The authors explore the challenges of variation across multiple levels: community, institution, program, and individual faculty members. The article concludes with recommendations to improve medical education by embracing the principles of continuous quality improvement to reduce the harmful effect of unwarranted variation.
- 21 July 2022
- Published by Oxford University Press (OUP) in Academic Medicine
- Vol. 97 (8) , 1128-1136
Open AccessWill Any Road Get You There? Examining Warranted and Unwarranted Variation in Medical Education
Undergraduate and graduate medical education have long embraced uniqueness and variability in curricular and assessment approaches. Some of this variability is justified (warranted or necessary variation), but a substantial portion represents unwarranted variation. A primary tenet of outcomes-based medical education is ensuring that all learners acquire essential competencies to be publicly accountable to meet societal needs. Unwarranted variation in curricular and assessment practices contributes to suboptimal and variable educational outcomes and, by extension, risks graduates delivering suboptimal health care quality. Medical education can use lessons from the decades of study on unwarranted variation in health care as part of efforts to continuously improve the quality of training programs. To accomplish this, medical educators will first need to recognize the difference between warranted and unwarranted variation in both clinical care and educational practices. Addressing unwarranted variation will require cooperation and collaboration between multiple levels of the health care and educational systems using a quality improvement mindset. These efforts at improvement should acknowledge that some aspects of variability are not scientifically informed and do not support desired outcomes or societal needs. This perspective examines the correlates of unwarranted variation of clinical care in medical education and the need to address the interdependency of unwarranted variation occurring between clinical and educational practices. The authors explore the challenges of variation across multiple levels: community, institution, program, and individual faculty members. The article concludes with recommendations to improve medical education by embracing the principles of continuous quality improvement to reduce the harmful effect of unwarranted variation.