Andrew E. Krumm
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Andrew E. Krumm has a total of 156 co-authors. Here are the top co-authors by number of shared publications:
- Brian C. George
Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, MI
14 shared publications - Kayla M. Marcotte
Department of Learning Health Sciences, University of Michigan, Ann Arbor, USA
11 shared publications - Alyssa A. Pradarelli
Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, MI
6 shared publications Brian C. George
6 shared publications- Rebecca Moreci
Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, MI
6 shared publications - Rebecca S. Gates
Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, USA
5 shared publications
Latest Publications
- 7 November 2026
- Published by Elsevier in Journal of Surgical Education
- Vol. 83 (12)
Evaluating the Feedback Quality of General Surgery Residents for Open Inguinal Hernia Repair
- 7 November 2026
- Published by Elsevier in Journal of Surgical Education
- Vol. 83 (12)
Evaluating the Feedback Quality of General Surgery Residents for Open Inguinal Hernia Repair
- Open Access
- 15 October 2026
- Published by Springer Nature in Global Surgical Education - Journal of the Association for Surgical Education
- Vol. 5 (1) , 200
Standard setting in the era of workplace-based assessment: a pilot exploration
Purpose: The use of workplace-based assessments (WBAs) in competency-based medical education continues to expand rapidly, and there are emerging methods to use these data to establish a trainee’s competence. However, establishing performance standards based on longitudinally collected operative performance data remains difficult, as existing standard-setting in medical education largely focuses on single tasks, single procedures, or written examinations. The purpose of this pilot study was to (1) assess the feasibility of adapting three established standard-setting methods to WBA ratings scored using the Society for Improving Medical Professional Learning (SIMPL) Operative Performance (OP) score and the convergence of the resulting cut scores in defining a general surgery graduation standard; and (2) identify the methodological and process considerations that could inform future WBA-based standard-setting efforts. Methods: A panel of 10 surgical educators participated in a two-hour virtual Benchmark Panel. Panelists were oriented to the SIMPL OR WBA tool and the OP score (range 372–628). Panelists applied three standard-setting methods to define a cut score corresponding to a “practice-ready” graduation standard for high-value, high-frequency procedures. Each panelist’s mean OP cut score per method was computed and compared using a one-way repeated-measures analysis of variance (ANOVA) with Holm-adjusted pairwise comparisons. The session transcript was analyzed qualitatively using interpretive description methodology to inductively identify themes to inform future iterations. Results: Mean (standard deviation) participant-level OP cut scores were 577 (30) for Angoff, 588 (17) for Bookmark, and 582 (19) for Construct Mapping. There was no association between standard setting method and participant-level OP cut scores [F(2, 18) = 0.63, p = 0.54, generalized η 2 = 0.045]. Holm-adjusted pairwise comparisons were also non-significant (all p > 0.05). Qualitative analysis surfaced themes related to the specific methods, including perceived high utility of Construct Mapping, and overall reflections on the standard-setting process, including the challenge of aligning on the mental model of a “minimally competent graduating trainee” and the difficulty in incorporating variation in case complexity and training experience. A proposed graduation standard of 582, anchored in the Construct Mapping method based on qualitative preference, corresponds to predicted probabilities of earning a practice-ready rating of 88–90% for laparoscopic appendectomy, 83–85% for laparoscopic cholecystectomy, and 76–80% for open partial colectomy. Conclusions: Three established standard-setting methods can be feasibly adapted to longitudinal operative performance WBA data and produced participant-level cut scores that were not statistically different. This convergence across methodologically independent approaches supports the proposed OP-score cut of 582 as an empirically grounded starting point for further multi-stakeholder consensus work. Method selection should be guided by methodological trade-offs in resource requirements, data dependencies, and perceived utility. These findings provide an initial framework for setting defensible, data-informed competency standards in the era of WBA.
- 15 October 2026
- Published by Springer Nature in Global Surgical Education - Journal of the Association for Surgical Education
- Vol. 5 (1) , 200
Open AccessStandard setting in the era of workplace-based assessment: a pilot exploration
Purpose: The use of workplace-based assessments (WBAs) in competency-based medical education continues to expand rapidly, and there are emerging methods to use these data to establish a trainee’s competence. However, establishing performance standards based on longitudinally collected operative performance data remains difficult, as existing standard-setting in medical education largely focuses on single tasks, single procedures, or written examinations. The purpose of this pilot study was to (1) assess the feasibility of adapting three established standard-setting methods to WBA ratings scored using the Society for Improving Medical Professional Learning (SIMPL) Operative Performance (OP) score and the convergence of the resulting cut scores in defining a general surgery graduation standard; and (2) identify the methodological and process considerations that could inform future WBA-based standard-setting efforts. Methods: A panel of 10 surgical educators participated in a two-hour virtual Benchmark Panel. Panelists were oriented to the SIMPL OR WBA tool and the OP score (range 372–628). Panelists applied three standard-setting methods to define a cut score corresponding to a “practice-ready” graduation standard for high-value, high-frequency procedures. Each panelist’s mean OP cut score per method was computed and compared using a one-way repeated-measures analysis of variance (ANOVA) with Holm-adjusted pairwise comparisons. The session transcript was analyzed qualitatively using interpretive description methodology to inductively identify themes to inform future iterations. Results: Mean (standard deviation) participant-level OP cut scores were 577 (30) for Angoff, 588 (17) for Bookmark, and 582 (19) for Construct Mapping. There was no association between standard setting method and participant-level OP cut scores [F(2, 18) = 0.63, p = 0.54, generalized η 2 = 0.045]. Holm-adjusted pairwise comparisons were also non-significant (all p > 0.05). Qualitative analysis surfaced themes related to the specific methods, including perceived high utility of Construct Mapping, and overall reflections on the standard-setting process, including the challenge of aligning on the mental model of a “minimally competent graduating trainee” and the difficulty in incorporating variation in case complexity and training experience. A proposed graduation standard of 582, anchored in the Construct Mapping method based on qualitative preference, corresponds to predicted probabilities of earning a practice-ready rating of 88–90% for laparoscopic appendectomy, 83–85% for laparoscopic cholecystectomy, and 76–80% for open partial colectomy. Conclusions: Three established standard-setting methods can be feasibly adapted to longitudinal operative performance WBA data and produced participant-level cut scores that were not statistically different. This convergence across methodologically independent approaches supports the proposed OP-score cut of 582 as an empirically grounded starting point for further multi-stakeholder consensus work. Method selection should be guided by methodological trade-offs in resource requirements, data dependencies, and perceived utility. These findings provide an initial framework for setting defensible, data-informed competency standards in the era of WBA.
- 1 June 2026
- Published by Journal of Graduate Medical Education in Journal of Graduate Medical Education
- Vol. 18 (3) , 334-350
Graduate Medical Education Data in the United States: Availability and Accessibility Across National Organizations
Background: Longitudinal, multi-organizational data have enormous potential for improving graduate medical education (GME) yet aggregating and sharing that data remain difficult. Objective: To determine what data is collected and how data is shared by US medical education organizations, as a next step toward expanded sharing of GME data. Methods: Ten multispecialty US medical education and licensing organizations that participated in the 2019 National Academies of Sciences, Engineering, and Medicine conference completed a pre-meeting data survey. Leaders from 9 of these organizations participated in semi-structured interviews conducted between January 2021 and May 2022. Content analysis of survey data and interview transcripts were used to construct a data inventory. Each of the 9 interviewed organizations reviewed and updated their inventory in January and February 2024. Inductive analyses of interview transcripts were conducted to identify additional themes. Results: The pre-meeting survey and interviews had a 100% (10 total) and 90% (9 of 10) response rate, respectively. Study participants collect a large amount of GME-related data, some of which is duplicative. None collect longer-term outcomes, including patient outcomes. There is also no common physician identifier used by all organizations, although Association of American Medical Colleges ID and National Provider Identifier have been adopted most widely. All organizations share data to varying degrees. Two themes emerged regarding ongoing efforts to expand data sharing: perceived benefits of a harmonized system and challenges to implementation. Conclusions: Data are increasingly available from organizations collecting national GME data in the United States.
- 1 June 2026
- Published by Journal of Graduate Medical Education in Journal of Graduate Medical Education
- Vol. 18 (3) , 334-350
Graduate Medical Education Data in the United States: Availability and Accessibility Across National Organizations
Background: Longitudinal, multi-organizational data have enormous potential for improving graduate medical education (GME) yet aggregating and sharing that data remain difficult. Objective: To determine what data is collected and how data is shared by US medical education organizations, as a next step toward expanded sharing of GME data. Methods: Ten multispecialty US medical education and licensing organizations that participated in the 2019 National Academies of Sciences, Engineering, and Medicine conference completed a pre-meeting data survey. Leaders from 9 of these organizations participated in semi-structured interviews conducted between January 2021 and May 2022. Content analysis of survey data and interview transcripts were used to construct a data inventory. Each of the 9 interviewed organizations reviewed and updated their inventory in January and February 2024. Inductive analyses of interview transcripts were conducted to identify additional themes. Results: The pre-meeting survey and interviews had a 100% (10 total) and 90% (9 of 10) response rate, respectively. Study participants collect a large amount of GME-related data, some of which is duplicative. None collect longer-term outcomes, including patient outcomes. There is also no common physician identifier used by all organizations, although Association of American Medical Colleges ID and National Provider Identifier have been adopted most widely. All organizations share data to varying degrees. Two themes emerged regarding ongoing efforts to expand data sharing: perceived benefits of a harmonized system and challenges to implementation. Conclusions: Data are increasingly available from organizations collecting national GME data in the United States.
- 1 April 2026
- Published by Journal of Graduate Medical Education in Journal of Graduate Medical Education
- Vol. 18 (2) , 216-224
An Exploration of Competency Assessment in Graduate Medical Education
- 1 April 2026
- Published by Journal of Graduate Medical Education in Journal of Graduate Medical Education
- Vol. 18 (2) , 216-224
An Exploration of Competency Assessment in Graduate Medical Education
- Open Access
- 2 February 2026
- Published by Springer Nature in Global Surgical Education - Journal of the Association for Surgical Education
- Vol. 5 (1)
Sound assessment in the era of AI
- 2 February 2026
- Published by Springer Nature in Global Surgical Education - Journal of the Association for Surgical Education
- Vol. 5 (1)
Open AccessSound assessment in the era of AI
- 2 January 2026
- Published by Elsevier in Journal of Surgical Education
- Vol. 83 (2) , 103822
EQIP 2023-2025: Increased Usability and Sustainability
- 2 January 2026
- Published by Elsevier in Journal of Surgical Education
- Vol. 83 (2) , 103822
EQIP 2023-2025: Increased Usability and Sustainability
- 15 May 2025
- Published by Wiley in BJU International
- Vol. 136 (4) , 633-639
What is lost in an average? Identifying distinct post‐radical prostatectomy functional recovery profiles
Objective: To describe, via latent variable mixture modelling, distinct post‐radical prostatectomy (RP) patient‐reported outcome (PRO) recovery profiles, which are positioned to complement currently disseminated statistical averages for shared decision‐making. Patients and Methods: Patients undergoing RP and completing the 26‐item Expanded Prostate Cancer Index Composite 12 months after surgery were identified from the Michigan Urological Surgery Improvement Collaborative data registry. Hierarchical cluster analysis and latent variable mixture modelling was applied to urinary incontinence (UI) and sexual function (SF) recovery scores, and final models chosen based on optimal performance. Results: A total of 3956 patients comprised the study cohort. Three distinct UI profiles were identified with prevalence of 49%, 37% and 14% from best to worst recovery, respectively. Four distinct SF profiles were identified with prevalence of 14%, 24%, 42%, and 20%, from best to worst recovery, respectively. The last two SF profiles had similar function scores but differed based on perception of function being bothersome. Limitations include incomplete PRO capture, which may introduce bias. Conclusions: We identify distinct UI and SF recovery profiles and their prevalence from a large, prospectively maintained registry, potentially improving interpretability of PRO data for decision making.
- 15 May 2025
- Published by Wiley in BJU International
- Vol. 136 (4) , 633-639
What is lost in an average? Identifying distinct post‐radical prostatectomy functional recovery profiles
Objective: To describe, via latent variable mixture modelling, distinct post‐radical prostatectomy (RP) patient‐reported outcome (PRO) recovery profiles, which are positioned to complement currently disseminated statistical averages for shared decision‐making. Patients and Methods: Patients undergoing RP and completing the 26‐item Expanded Prostate Cancer Index Composite 12 months after surgery were identified from the Michigan Urological Surgery Improvement Collaborative data registry. Hierarchical cluster analysis and latent variable mixture modelling was applied to urinary incontinence (UI) and sexual function (SF) recovery scores, and final models chosen based on optimal performance. Results: A total of 3956 patients comprised the study cohort. Three distinct UI profiles were identified with prevalence of 49%, 37% and 14% from best to worst recovery, respectively. Four distinct SF profiles were identified with prevalence of 14%, 24%, 42%, and 20%, from best to worst recovery, respectively. The last two SF profiles had similar function scores but differed based on perception of function being bothersome. Limitations include incomplete PRO capture, which may introduce bias. Conclusions: We identify distinct UI and SF recovery profiles and their prevalence from a large, prospectively maintained registry, potentially improving interpretability of PRO data for decision making.
- 24 February 2025
- Published by Springer Nature in Advances in Health Sciences Education
- Vol. 30 (1) , 87-101
Ethical considerations of using learning analytics in medical education: a critical review
Learning analytics are increasingly used in medical education to analyze data and make decisions about learners’ abilities. While there are many potential benefits of using learning analytics to drive improvement in medical education, there are also ethical concerns surrounding how this may affect learners and their patients. We conducted a critical review of studies that use learning analytics and big data within medical education. Using guidelines established by Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA), relevant articles were identified in MEDLINE (PubMed) and SocINDEX databases from inception to April 2021. Detailed data abstraction was performed across studies to identify current uses of learning analytics and identify potential ethical concerns. Eighteen articles met the search criteria. Our analysis identified the use of learning analytics and big data in four aspects of medical education: (1) the learning process and pedagogy; (2) retrospective assessment; (3) prospective assessment; and (4) improvement of education. We identified some ethical concerns surrounding the use of learning analytics and big data, including the (1) trustworthiness of data; (2) reliability of methodology; (3) privacy, confidentiality, and management of data; and (4) labeling of learners as “problematic.” Using Beauchamp and Childress’s biomedical ethics as a framework, we identified potential consequences of using learning analytics for learners within the principles of beneficence, nonmaleficence, autonomy, and justice. As learning analytics becomes more widespread in medical education, examining and mitigating potential harm towards learners is imperative.
- 24 February 2025
- Published by Springer Nature in Advances in Health Sciences Education
- Vol. 30 (1) , 87-101
Ethical considerations of using learning analytics in medical education: a critical review
Learning analytics are increasingly used in medical education to analyze data and make decisions about learners’ abilities. While there are many potential benefits of using learning analytics to drive improvement in medical education, there are also ethical concerns surrounding how this may affect learners and their patients. We conducted a critical review of studies that use learning analytics and big data within medical education. Using guidelines established by Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA), relevant articles were identified in MEDLINE (PubMed) and SocINDEX databases from inception to April 2021. Detailed data abstraction was performed across studies to identify current uses of learning analytics and identify potential ethical concerns. Eighteen articles met the search criteria. Our analysis identified the use of learning analytics and big data in four aspects of medical education: (1) the learning process and pedagogy; (2) retrospective assessment; (3) prospective assessment; and (4) improvement of education. We identified some ethical concerns surrounding the use of learning analytics and big data, including the (1) trustworthiness of data; (2) reliability of methodology; (3) privacy, confidentiality, and management of data; and (4) labeling of learners as “problematic.” Using Beauchamp and Childress’s biomedical ethics as a framework, we identified potential consequences of using learning analytics for learners within the principles of beneficence, nonmaleficence, autonomy, and justice. As learning analytics becomes more widespread in medical education, examining and mitigating potential harm towards learners is imperative.
- Open Access
- 26 December 2024
- Published by Ubiquity Press, Ltd. in Perspectives on Medical Education
- Vol. 13 (1) , 646-653
Proximity to Practice: The Role of Technology in the Next Era of Assessment
Perspectives on Medical Education’s mission is to support and enrich collaborative scholarship between education researchers and clinical educators, and to advance new knowledge regarding clinical education practices. The journal is the official journal of the The Netherlands Association of Medical Education (NVMO), which sponsors the journal as an Open Access journal that is free to read immediately upon publication. To publish in the journal, authors are required to pay an author processing charge of £1026 upon publication. The journal is indexed in key databases, including MEDLINE, Web of Science, Google Scholar, and Scopus and has received the DOAJ seal. In 2023, the journal was awarded an impact factor of 4.8. The journal is overseen by the Editor-in-Chief Professor Erik Driessen, PhD. He is provided by guidance by an international editorial board of leading scholars with members representing the different parental disciplines in the field of medical education, e.g. clinicians, social scientists, biomedical scientists, statisticians, and linguists.
- 26 December 2024
- Published by Ubiquity Press, Ltd. in Perspectives on Medical Education
- Vol. 13 (1) , 646-653
Open AccessProximity to Practice: The Role of Technology in the Next Era of Assessment
Perspectives on Medical Education’s mission is to support and enrich collaborative scholarship between education researchers and clinical educators, and to advance new knowledge regarding clinical education practices. The journal is the official journal of the The Netherlands Association of Medical Education (NVMO), which sponsors the journal as an Open Access journal that is free to read immediately upon publication. To publish in the journal, authors are required to pay an author processing charge of £1026 upon publication. The journal is indexed in key databases, including MEDLINE, Web of Science, Google Scholar, and Scopus and has received the DOAJ seal. In 2023, the journal was awarded an impact factor of 4.8. The journal is overseen by the Editor-in-Chief Professor Erik Driessen, PhD. He is provided by guidance by an international editorial board of leading scholars with members representing the different parental disciplines in the field of medical education, e.g. clinicians, social scientists, biomedical scientists, statisticians, and linguists.
- Open Access
- 20 November 2024
- Published by Ubiquity Press, Ltd. in Perspectives on Medical Education
- Vol. 13 (1) , 553-560
Digital Evidence: Revisiting Assumptions at the Intersection of Technology and Assessment
The increasing use of technology in health care and health professions education is an invitation to examine how digital sources of evidence are used in making assessment claims. In this paper, we describe how four sets of terms—primary and secondary data; structured and unstructured data; development and use; and deterministic and generative—can aid in examining how data from digital sources are used in evaluating what learners know and can do. Drawing on multiple examples, this paper shows how the four sets of terms can help both developers and users of technology-based assessment systems.
- 20 November 2024
- Published by Ubiquity Press, Ltd. in Perspectives on Medical Education
- Vol. 13 (1) , 553-560
Open AccessDigital Evidence: Revisiting Assumptions at the Intersection of Technology and Assessment
The increasing use of technology in health care and health professions education is an invitation to examine how digital sources of evidence are used in making assessment claims. In this paper, we describe how four sets of terms—primary and secondary data; structured and unstructured data; development and use; and deterministic and generative—can aid in examining how data from digital sources are used in evaluating what learners know and can do. Drawing on multiple examples, this paper shows how the four sets of terms can help both developers and users of technology-based assessment systems.