Katherine Ann Moreau
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Katherine Ann Moreau has a total of 54 co-authors. Here are the top co-authors by number of shared publications:
Kaylee Eady
15 shared publications- Kaylee Eady
University of Ottawa
10 shared publications - Catherine M. Giroux
University of Ottawa
4 shared publications - Lindsey Sikora
Health Sciences Library, University of Ottawa, Ottawa, Ontario, Canada
3 shared publications - Alison B. Flynn
University of Ottawa
2 shared publications - Danny Yu Jia Ke
Queen’s University School of Medicine, Kingston, ON, Canada
2 shared publications
Latest Publications
- Open Access
- 15 December 2025
- Published by Center for Open Science in EdArXiv
Searching for Patient Involvement in Obstetrics and Gynecology Education and Training; A Scoping Review
Background: The active involvement of patients in Health Professions Education (HPE) fosters partnerships and enhances patient-centred care. While reviews have explored this topic broadly, patient involvement in Obstetrics and Gynecology (OB/GYN) education and training, an area of care that addresses sensitive issues, remains poorly understood. Synthesized evidence is crucial in guiding research priorities.Objective: The purpose of this scoping study was to comprehensively synthesize English and French evidence on patient involvement in OB/GYN education and training. The following research questions were addressed: (1) How is patient involvement in OB/GYN education and training conceptualized? (2) What is the nature and state of research on this topic? (3) What are priority areas for future research?Methods: With the support of a Research Librarian, we developed and executed search strategies in EMBASE, Education Source, Academic Search Complete, Web of Science, MEDLINE, ERIC, APA PsycINFO, and CINAHL, utilizing Covidence to facilitate screening and publication selection. Included publications detail patient involvement in OB/GYN education and training within medicine, nursing, and midwifery, published in English and French. We conducted a complementary search of reference lists and relevant educational and professional websites. We iteratively developed and piloted a data extraction form in Excel, extracting data from all included sources.Results: After screening 3,085 titles and abstracts and reviewing 147 full texts, 34 publications were included, with 11 added through reference list reviews, for a total of 45. Patient involvement was primarily used as a tool to enhance HPE programming, ensuring content meets patient needs, and supporting competency development. Most examples came from midwifery, followed by medicine and nursing. Patients were typically positioned as information sources, sharing their lived experience. A key gap was inconsistent reporting, with many studies lacking essential contextual details about the nature of involvement. Conclusions: Patient involvement remains a passive focus in the field of OB/GYN and requires more targeted research. Particularly, the field would benefit from clear reporting standards and a shared framework for evaluating involvement that captures its contextual complexity. Keywords: health professions education, obstetrics, gynecology, patient involvement
- 15 December 2025
- Published by Center for Open Science in EdArXiv
Open AccessSearching for Patient Involvement in Obstetrics and Gynecology Education and Training; A Scoping Review
Background: The active involvement of patients in Health Professions Education (HPE) fosters partnerships and enhances patient-centred care. While reviews have explored this topic broadly, patient involvement in Obstetrics and Gynecology (OB/GYN) education and training, an area of care that addresses sensitive issues, remains poorly understood. Synthesized evidence is crucial in guiding research priorities.Objective: The purpose of this scoping study was to comprehensively synthesize English and French evidence on patient involvement in OB/GYN education and training. The following research questions were addressed: (1) How is patient involvement in OB/GYN education and training conceptualized? (2) What is the nature and state of research on this topic? (3) What are priority areas for future research?Methods: With the support of a Research Librarian, we developed and executed search strategies in EMBASE, Education Source, Academic Search Complete, Web of Science, MEDLINE, ERIC, APA PsycINFO, and CINAHL, utilizing Covidence to facilitate screening and publication selection. Included publications detail patient involvement in OB/GYN education and training within medicine, nursing, and midwifery, published in English and French. We conducted a complementary search of reference lists and relevant educational and professional websites. We iteratively developed and piloted a data extraction form in Excel, extracting data from all included sources.Results: After screening 3,085 titles and abstracts and reviewing 147 full texts, 34 publications were included, with 11 added through reference list reviews, for a total of 45. Patient involvement was primarily used as a tool to enhance HPE programming, ensuring content meets patient needs, and supporting competency development. Most examples came from midwifery, followed by medicine and nursing. Patients were typically positioned as information sources, sharing their lived experience. A key gap was inconsistent reporting, with many studies lacking essential contextual details about the nature of involvement. Conclusions: Patient involvement remains a passive focus in the field of OB/GYN and requires more targeted research. Particularly, the field would benefit from clear reporting standards and a shared framework for evaluating involvement that captures its contextual complexity. Keywords: health professions education, obstetrics, gynecology, patient involvement
- Open Access
- 1 January 2025
- Published by Wiley in The Clinical Teacher
- Vol. 22 (4) , e70129
Perspectives on Involving Patients in the Teaching and Assessment of Entrustable Professional Activities in Competence by Design
Background: Involving patients in teaching and assessing entrustable professional activities (EPAs) within competency‐based medical education (CBME) enhances the authenticity of postgraduate medical education and helps develop learners' skills. However, CBME, including Competence by Design in Canada, lacks clear guidance on involving patients. This study examines faculty members' perspectives on involving patients in teaching and assessing EPAs. Methods: We conducted semistructured interviews with 25 faculty members from 14 Canadian medical schools across eight residency specialties. We asked participants how they envisioned patients being involved in teaching and assessing EPAs during EPA creation, how they thought patients could contribute to teaching and assessing EPAs, and what they perceived as barriers. We analysed the data thematically. Findings: Faculty members view patients as subjects for teaching EPAs rather than as teachers of EPAs. However, they noted that patients can teach aspects of EPAs through storytelling, if provided the opportunity. They recognized the value of patients as assessors, particularly in formative assessments of learners' nontechnical skills embedded within EPAs. Nevertheless, they expressed concerns about patient involvement in summative assessments and a lack of EPAs focusing on skills that patients can assess. They noted several barriers to involving patients in teaching and assessing EPAs. Conclusion: Our study underscores the pressing need for change and the crucial role of more explicit guidance on involving patients in teaching and assessing EPAs. This is not just a matter of academic interest, but a step towards enhancing the quality of medical education.
- 1 January 2025
- Published by Wiley in The Clinical Teacher
- Vol. 22 (4) , e70129
Open AccessPerspectives on Involving Patients in the Teaching and Assessment of Entrustable Professional Activities in Competence by Design
Background: Involving patients in teaching and assessing entrustable professional activities (EPAs) within competency‐based medical education (CBME) enhances the authenticity of postgraduate medical education and helps develop learners' skills. However, CBME, including Competence by Design in Canada, lacks clear guidance on involving patients. This study examines faculty members' perspectives on involving patients in teaching and assessing EPAs. Methods: We conducted semistructured interviews with 25 faculty members from 14 Canadian medical schools across eight residency specialties. We asked participants how they envisioned patients being involved in teaching and assessing EPAs during EPA creation, how they thought patients could contribute to teaching and assessing EPAs, and what they perceived as barriers. We analysed the data thematically. Findings: Faculty members view patients as subjects for teaching EPAs rather than as teachers of EPAs. However, they noted that patients can teach aspects of EPAs through storytelling, if provided the opportunity. They recognized the value of patients as assessors, particularly in formative assessments of learners' nontechnical skills embedded within EPAs. Nevertheless, they expressed concerns about patient involvement in summative assessments and a lack of EPAs focusing on skills that patients can assess. They noted several barriers to involving patients in teaching and assessing EPAs. Conclusion: Our study underscores the pressing need for change and the crucial role of more explicit guidance on involving patients in teaching and assessing EPAs. This is not just a matter of academic interest, but a step towards enhancing the quality of medical education.
- Open Access
- 1 January 2025
- Published by SAGE Publications in Journal of Medical Education and Curricular Development
- Vol. 12
Program Evaluation in Competence by Design: A Mixed-Methods Study
BACKGROUND: The evaluation of Competence by Design (CBD) residency programs is crucial for enhancing program effectiveness. However, literature on evaluating CBD programs is limited. We conducted a 2-phase mixed-methods study to (a) assess the extent of program evaluation activities in CBD residency programs in Canada, (b) explore reasons for engaging or not engaging in these activities, (c) examine how CBD programs are conducting program evaluations, and (d) identify ways to build capacity for program evaluation. METHODS: Phase 1 involved surveying 149 program directors from specialty/subspecialty programs that transitioned to CBD between 2017 and 2020. We calculated descriptive statistics for 22 closed-ended survey items. Phase 2 comprised interviews with a subset of program directors from Phase 1. Data analysis followed a 3-step iterative process: data condensation, data display, and drawing and verifying conclusions. RESULTS: In Phase 1, we received 149 responses, with a 33.5% response rate. Of these, 127 (85.2%) indicated their programs engage in evaluation, while 22 (14.8%) do not. Among the 127 programs that engage in evaluation, 29 (22.8%) frequently or always develop evaluation questions, and 23 (18.1%) design evaluation proposals/plans. Reasons for engaging in evaluation included decision-making and stimulating changes in educational practices. Conversely, reasons for not engaging included lack of knowledge, personnel, and funding. In Phase 2, 15 program directors were interviewed. They reported that CBD programs face challenges such as limited resources and buy-in, rely on ad hoc evaluation methods, and use a team-based evaluation format. To enhance evaluation capacities, interviewees suggested (a) developing expertise in program evaluation, (b) acquiring evaluation resources, and (c) advocating for clear evaluation expectations. CONCLUSIONS: Most CBD residency programs are engaged in program evaluations, but the quality is often questionable. To fully realize the potential of program evaluation, CBD programs need additional resources and support to improve evaluation practices and outcomes.
- 1 January 2025
- Published by SAGE Publications in Journal of Medical Education and Curricular Development
- Vol. 12
Open AccessProgram Evaluation in Competence by Design: A Mixed-Methods Study
BACKGROUND: The evaluation of Competence by Design (CBD) residency programs is crucial for enhancing program effectiveness. However, literature on evaluating CBD programs is limited. We conducted a 2-phase mixed-methods study to (a) assess the extent of program evaluation activities in CBD residency programs in Canada, (b) explore reasons for engaging or not engaging in these activities, (c) examine how CBD programs are conducting program evaluations, and (d) identify ways to build capacity for program evaluation. METHODS: Phase 1 involved surveying 149 program directors from specialty/subspecialty programs that transitioned to CBD between 2017 and 2020. We calculated descriptive statistics for 22 closed-ended survey items. Phase 2 comprised interviews with a subset of program directors from Phase 1. Data analysis followed a 3-step iterative process: data condensation, data display, and drawing and verifying conclusions. RESULTS: In Phase 1, we received 149 responses, with a 33.5% response rate. Of these, 127 (85.2%) indicated their programs engage in evaluation, while 22 (14.8%) do not. Among the 127 programs that engage in evaluation, 29 (22.8%) frequently or always develop evaluation questions, and 23 (18.1%) design evaluation proposals/plans. Reasons for engaging in evaluation included decision-making and stimulating changes in educational practices. Conversely, reasons for not engaging included lack of knowledge, personnel, and funding. In Phase 2, 15 program directors were interviewed. They reported that CBD programs face challenges such as limited resources and buy-in, rely on ad hoc evaluation methods, and use a team-based evaluation format. To enhance evaluation capacities, interviewees suggested (a) developing expertise in program evaluation, (b) acquiring evaluation resources, and (c) advocating for clear evaluation expectations. CONCLUSIONS: Most CBD residency programs are engaged in program evaluations, but the quality is often questionable. To fully realize the potential of program evaluation, CBD programs need additional resources and support to improve evaluation practices and outcomes.
- Open Access
- 25 November 2024
- Published by The Association of Faculties of Medicine of Canada in Canadian Medical Education Journal
- Vol. 15 (6) , 93-94
Beyond the classroom: enhancing graduate health professions education programs for non-clinicians
- 25 November 2024
- Published by The Association of Faculties of Medicine of Canada in Canadian Medical Education Journal
- Vol. 15 (6) , 93-94
Open AccessBeyond the classroom: enhancing graduate health professions education programs for non-clinicians
- Open Access
- 14 November 2024
- Published by The Beryl Institute in Patient Experience Journal
- Vol. 11 (3) , 29-36
Patient, Family, and Health Professional Perspectives of How Families are Involved in Adult Inpatient Traumatic Brain Injury Rehabilitation
Objective: We aimed to answer the following research question: From the perspectives of patients, family members, and health professionals, how are families involved in the rehabilitation of adult patients with Traumatic Brain Injury at an inpatient Acquired Brain Injury service within an adult rehabilitation centre? Methods: We used an interpretive qualitative approach, from a constructivist view. This approach allowed us to understand and describe family involvement from the perspectives of patients with Traumatic Brain Injury, family members, and health professionals. We conducted one-on-one interviews with patients, family members, and health professionals. We used a three-step analysis method: data reduction, data display, and conclusions/verifications. Results: Six patients, four family members, and ten health professionals participated. Participants described how families are involved in supporting and encouraging the patient, seeking and exchanging information, making decisions, and care and therapy. Conclusion: Overall, patients and family members perceived family involvement as critical, while health professionals viewed it as not always necessary. This study also revealed that patients and family members shared similar understandings of family involvement but had different understandings than health professionals. Recognizing that the adult population and rehabilitation processes are unique and complex, we need to understand how to support family involvement, through patient- and family-centered care, in this type of setting. We also need to understand how to better facilitate patient-family-health professional collaboration.
- 14 November 2024
- Published by The Beryl Institute in Patient Experience Journal
- Vol. 11 (3) , 29-36
Open AccessPatient, Family, and Health Professional Perspectives of How Families are Involved in Adult Inpatient Traumatic Brain Injury Rehabilitation
Objective: We aimed to answer the following research question: From the perspectives of patients, family members, and health professionals, how are families involved in the rehabilitation of adult patients with Traumatic Brain Injury at an inpatient Acquired Brain Injury service within an adult rehabilitation centre? Methods: We used an interpretive qualitative approach, from a constructivist view. This approach allowed us to understand and describe family involvement from the perspectives of patients with Traumatic Brain Injury, family members, and health professionals. We conducted one-on-one interviews with patients, family members, and health professionals. We used a three-step analysis method: data reduction, data display, and conclusions/verifications. Results: Six patients, four family members, and ten health professionals participated. Participants described how families are involved in supporting and encouraging the patient, seeking and exchanging information, making decisions, and care and therapy. Conclusion: Overall, patients and family members perceived family involvement as critical, while health professionals viewed it as not always necessary. This study also revealed that patients and family members shared similar understandings of family involvement but had different understandings than health professionals. Recognizing that the adult population and rehabilitation processes are unique and complex, we need to understand how to support family involvement, through patient- and family-centered care, in this type of setting. We also need to understand how to better facilitate patient-family-health professional collaboration.
- Open Access
- 2 August 2024
- Published by Ubiquity Press, Ltd. in Perspectives on Medical Education
- Vol. 13 (1) , 417-422
An Innovative Course on Involving Patients in Health Professions Education
Perspectives on Medical Education’s mission is to support and enrich collaborative scholarship between education researchers and clinical educators, and to advance new knowledge regarding clinical education practices. The journal is the official journal of the The Netherlands Association of Medical Education (NVMO), which sponsors the journal as an Open Access journal that is free to read immediately upon publication. To publish in the journal, authors are required to pay an author processing charge of £996 upon publication. The journal is indexed in key databases, including MEDLINE, Web of Science, Google Scholar, and Scopus and has received the DOAJ seal. In 2023, the journal was awarded an impact factor of 4.8. The journal is overseen by the Editor-in-Chief Professor Erik Driessen, PhD. He is provided by guidance by an international editorial board of leading scholars with members representing the different parental disciplines in the field of medical education, e.g. clinicians, social scientists, biomedical scientists, statisticians, and linguists.
- 2 August 2024
- Published by Ubiquity Press, Ltd. in Perspectives on Medical Education
- Vol. 13 (1) , 417-422
Open AccessAn Innovative Course on Involving Patients in Health Professions Education
Perspectives on Medical Education’s mission is to support and enrich collaborative scholarship between education researchers and clinical educators, and to advance new knowledge regarding clinical education practices. The journal is the official journal of the The Netherlands Association of Medical Education (NVMO), which sponsors the journal as an Open Access journal that is free to read immediately upon publication. To publish in the journal, authors are required to pay an author processing charge of £996 upon publication. The journal is indexed in key databases, including MEDLINE, Web of Science, Google Scholar, and Scopus and has received the DOAJ seal. In 2023, the journal was awarded an impact factor of 4.8. The journal is overseen by the Editor-in-Chief Professor Erik Driessen, PhD. He is provided by guidance by an international editorial board of leading scholars with members representing the different parental disciplines in the field of medical education, e.g. clinicians, social scientists, biomedical scientists, statisticians, and linguists.
- Open Access
- 29 February 2024
- Published by University of Western Ontario, Western Libraries in The Canadian Journal for the Scholarship of Teaching and Learning
- Vol. 15 (1)
What Drives Nursing Students’ Social Media-Related Decisions in their Learning Practices? A Survey of an Ontario School of Nursing
Social media can provide a tool for nursing students to consolidate their formal and informal learning experiences. The objective of this study was to explore what drives nursing students’ decisions to use social media in their formal and informal learning. This study surveyed 220 nursing students at one Ontario School of Nursing. IBM SPSS (v. 24) was used to calculate frequency counts, Chi-Square Tests for Independence, and Spearman Correlations. A modified directed content analysis was conducted on the open-ended response data. Numerous drivers affect nursing students’ decisions to use social media for learning purposes. These include: 1) their nursing programs; 2) their professors’ attitudes towards social media; 3) their age and perceived levels of experience using social media; 4) the convenience of accessing learning content online; and 5) barriers like privacy, quality and reliability of social media-based learning content, and the potential for distraction. As a result of these findings, there is a strong rationale to further explore the ways in which social media can be used in both formal and informal nursing education.
- 29 February 2024
- Published by University of Western Ontario, Western Libraries in The Canadian Journal for the Scholarship of Teaching and Learning
- Vol. 15 (1)
Open AccessWhat Drives Nursing Students’ Social Media-Related Decisions in their Learning Practices? A Survey of an Ontario School of Nursing
Social media can provide a tool for nursing students to consolidate their formal and informal learning experiences. The objective of this study was to explore what drives nursing students’ decisions to use social media in their formal and informal learning. This study surveyed 220 nursing students at one Ontario School of Nursing. IBM SPSS (v. 24) was used to calculate frequency counts, Chi-Square Tests for Independence, and Spearman Correlations. A modified directed content analysis was conducted on the open-ended response data. Numerous drivers affect nursing students’ decisions to use social media for learning purposes. These include: 1) their nursing programs; 2) their professors’ attitudes towards social media; 3) their age and perceived levels of experience using social media; 4) the convenience of accessing learning content online; and 5) barriers like privacy, quality and reliability of social media-based learning content, and the potential for distraction. As a result of these findings, there is a strong rationale to further explore the ways in which social media can be used in both formal and informal nursing education.
- Open Access
- 16 October 2023
- Published by Springer Nature in Research Square
Program Evaluation Activities in Competence by Design: A Survey of Specialty/Subspecialty Program Directors
Background The evaluation of Competence by Design (CBD) residency programs is essential for improving program effectiveness. There is limited literature on the evaluation of CBD programs. We investigated the extent to which program evaluation activities are occurring in CBD residency programs in Canada and the reasons why these programs are engaging or not engaging in them. Methods We surveyed program directors whose programs transitioned to CBD. We calculated descriptive statistics for the 22 closed-ended survey items. Results We obtained 149 responses (response rate 33.5%). Of the 149 respondents, 127 (85.2%) indicated that their programs do engage in evaluation while 22 (14.8%) indicated that their programs do not. Of the 127 whose programs do engage in evaluation, 29 (22.8%) reported that their programs frequently or always develop evaluation questions and 23 (18.1%) noted that their programs design evaluation proposals/plans. Reasons for engaging in evaluation included: to make decisions about the program, and to stimulate changes in educational practices. Reasons for not engaging in evaluation comprised: no knowledge on how to do it, no personnel to do evaluation, and no funding to do it. Conclusions While most CBD programs are doing program evaluation the quality of it is questionable.
- 16 October 2023
- Published by Springer Nature in Research Square
Open AccessProgram Evaluation Activities in Competence by Design: A Survey of Specialty/Subspecialty Program Directors
Background The evaluation of Competence by Design (CBD) residency programs is essential for improving program effectiveness. There is limited literature on the evaluation of CBD programs. We investigated the extent to which program evaluation activities are occurring in CBD residency programs in Canada and the reasons why these programs are engaging or not engaging in them. Methods We surveyed program directors whose programs transitioned to CBD. We calculated descriptive statistics for the 22 closed-ended survey items. Results We obtained 149 responses (response rate 33.5%). Of the 149 respondents, 127 (85.2%) indicated that their programs do engage in evaluation while 22 (14.8%) indicated that their programs do not. Of the 127 whose programs do engage in evaluation, 29 (22.8%) reported that their programs frequently or always develop evaluation questions and 23 (18.1%) noted that their programs design evaluation proposals/plans. Reasons for engaging in evaluation included: to make decisions about the program, and to stimulate changes in educational practices. Reasons for not engaging in evaluation comprised: no knowledge on how to do it, no personnel to do evaluation, and no funding to do it. Conclusions While most CBD programs are doing program evaluation the quality of it is questionable.
- Open Access
- 15 September 2023
- Published by The Evaluation Center at Western Michigan University in Journal of Multidisciplinary Evaluation
- Vol. 19 (45) , 67-80
Excessive Evaluation Anxiety (XEA): The Last Two Decades
Background: Excessive evaluation anxiety (XEA) refers to disproportionate or increased evaluation anxiety among those affected by evaluation (e.g., stakeholders) characterized by the sole presence of negative consequences. It can compromise evaluator-stakeholder relationships, presenting as a barrier for program evaluation. Moreover, XEA can both cause and be caused by resistance to evaluation, which is an interrelated topic that shares many common causes, characteristics, and mitigation strategies. The participatory and interactive nature of modern evaluation approaches can exacerbate the presence of XEA. However, researchers have not explored the current state of literature on XEA. Purpose: To explore the current state of the literature on XEA over the past 20 years. Setting: Not applicable. Intervention: Not applicable. Research Design: Literature review. Data Collection and Analysis: We conducted a literature search of Academic Search Complete, Web of Science, and Scopus. We complemented the database search by a journal search of the American Journal of Evaluation, Evaluation, and the Canadian Journal of Program Evaluation. We then conducted a thematic analysis of the articles that met the inclusion criteria. Findings: Upon review of the articles, we identified four main themes in the literature related to XEA. Specifically, XEA: leads to poor evaluator-stakeholder relationships; is influenced by cultural factors; can be mitigated through the development of interpersonal skills; and can be mitigated through a systematic and evidence-based approach to evaluation.
- 15 September 2023
- Published by The Evaluation Center at Western Michigan University in Journal of Multidisciplinary Evaluation
- Vol. 19 (45) , 67-80
Open AccessExcessive Evaluation Anxiety (XEA): The Last Two Decades
Background: Excessive evaluation anxiety (XEA) refers to disproportionate or increased evaluation anxiety among those affected by evaluation (e.g., stakeholders) characterized by the sole presence of negative consequences. It can compromise evaluator-stakeholder relationships, presenting as a barrier for program evaluation. Moreover, XEA can both cause and be caused by resistance to evaluation, which is an interrelated topic that shares many common causes, characteristics, and mitigation strategies. The participatory and interactive nature of modern evaluation approaches can exacerbate the presence of XEA. However, researchers have not explored the current state of literature on XEA. Purpose: To explore the current state of the literature on XEA over the past 20 years. Setting: Not applicable. Intervention: Not applicable. Research Design: Literature review. Data Collection and Analysis: We conducted a literature search of Academic Search Complete, Web of Science, and Scopus. We complemented the database search by a journal search of the American Journal of Evaluation, Evaluation, and the Canadian Journal of Program Evaluation. We then conducted a thematic analysis of the articles that met the inclusion criteria. Findings: Upon review of the articles, we identified four main themes in the literature related to XEA. Specifically, XEA: leads to poor evaluator-stakeholder relationships; is influenced by cultural factors; can be mitigated through the development of interpersonal skills; and can be mitigated through a systematic and evidence-based approach to evaluation.
- Open Access
- 1 February 2023
- Published by Journal of Graduate Medical Education in Journal of Graduate Medical Education
- Vol. 15 (1) , 15-18
Program Evaluation Use in Graduate Medical Education
It is common to complete evaluations of graduate medical education (GME) programs, present them at conferences, publish them in peer-reviewed journals, add them to curricula vitae (CVs), and then move on without using them to enact changes in the programs themselves. Such actions may reflect the reality that many individuals perceive and conduct program evaluations as if they were research.1 While research and program evaluation use similar methods, they have distinct purposes, timelines, audiences, and most notably, intended uses.2 Evaluations of GME programs need to be used to, for example, inform program decisions and modifications, grow program stakeholders' knowledge, stimulate organizational culture changes, or improve the quality of training.1,3,4 They need to be more than intellectual exercises resulting in accomplishments listed on CVs.5 As such, we emphasize that evaluation use is an essential consequence of program evaluation. Those involved in program evaluation should discuss it and maintain its prominence at the onset of every evaluation. We also promote the adage “use-it-or-lose-it” to stress timely program evaluation use. Yet the literature on program evaluation in GME often neglects to discuss use, including how selected evaluation approaches can influence evaluation use.1,6,7 In this article, we explain evaluation use by describing both the use of evaluation findings and process use (ie, changes resulting from engagement in the evaluation process itself).1,8 We also suggest strategies, including evaluation approaches, that faculty can use to increase evaluation use in GME.The 3 categories of use of evaluation findings are instrumental, conceptual, and symbolic. Instrumental use refers to instances where stakeholders use evaluation findings to take direct actions (eg, improvements, changes, terminations) in a program.9 For example, evaluation findings show that residents in a GME program are struggling to complete their research projects. Using the findings, the GME team implements new research training activities to assist residents in the completion of their projects. Conceptual use describes occurrences where stakeholders use evaluation findings to evolve their understandings of a program but do not take direct actions based on these findings.4 For instance, the GME team acknowledges the findings that residents are struggling to complete their research projects. These findings inform their understanding of why residents are not attending academic conferences to present their research. Lastly, symbolic use occurs when stakeholders use the sheer existence of a completed evaluation to comply with reporting requirements or justify a previously made program action.4 For example, the funding university requires the GME program to complete an evaluation to retain funding for residents' research projects. The GME team completes an evaluation and presents the report to the university. Alternatively, before the evaluation, the GME program hired a research assistant to help residents with their research projects and the subsequent evaluation findings are used to justify the hiring of the research assistant. In GME, we emphasize instrumental use, as this form of use leads to actions that can improve programs. However, the use of evaluation findings is typically a short-term consequence of evaluation because these findings are relevant only within a specific and limited timeframe (ie, use-it-or-lose it).On the other hand, process use can have ongoing influence on individuals, programs, and organizations. It recognizes that evaluation processes themselves can affect attitudes, thought processes, and behaviors.10 Process use recognizes stakeholders' learning advancements from their involvement in an evaluation as well as the effects of evaluation processes on program functioning and organizational culture.11 Process use does not require changes to a program or direct actions because of evaluation findings. There are 6 types of process use which we illustrate with examples:When stakeholders are involved in evaluation processes, they enter an evaluation culture and learn how to think and look at things through an evaluative lens. They can also use the knowledge and skills (eg, evaluation knowledge, methodological and facilitation skills) they develop to strengthen their organization's abilities to design, implement, interpret, and use evaluations and thereby build their organization's evaluation capacity. In this sense, process use is valuable throughout and following an evaluation and in various GME settings regardless of the evaluation findings or recommendations.12The Table presents strategies that faculty involved in program evaluation can employ to increase evaluation use.In closing, it is imperative to remember that evaluation use, especially process use, can occur throughout a program evaluation rather than simply at its conclusion.10 Evaluation use can start at the planning stage and continue well beyond a presentation or publication of an evaluation. Program evaluators need a use-it-or-lose-it perspective throughout the evaluation process to maximize improvements to training. This perspective will maintain stakeholders' faith in the value of evaluation, as they witness that evaluation efforts lead to timely, actionable findings and processes. Ultimately, we must embrace evaluation use to ensure that all stakeholders and programs, not only conference attendees, readers of peer-reviewed journals, or our CVs, witness the consequences (both positive and negative) of program evaluation.
- 1 February 2023
- Published by Journal of Graduate Medical Education in Journal of Graduate Medical Education
- Vol. 15 (1) , 15-18
Open AccessProgram Evaluation Use in Graduate Medical Education
It is common to complete evaluations of graduate medical education (GME) programs, present them at conferences, publish them in peer-reviewed journals, add them to curricula vitae (CVs), and then move on without using them to enact changes in the programs themselves. Such actions may reflect the reality that many individuals perceive and conduct program evaluations as if they were research.1 While research and program evaluation use similar methods, they have distinct purposes, timelines, audiences, and most notably, intended uses.2 Evaluations of GME programs need to be used to, for example, inform program decisions and modifications, grow program stakeholders' knowledge, stimulate organizational culture changes, or improve the quality of training.1,3,4 They need to be more than intellectual exercises resulting in accomplishments listed on CVs.5 As such, we emphasize that evaluation use is an essential consequence of program evaluation. Those involved in program evaluation should discuss it and maintain its prominence at the onset of every evaluation. We also promote the adage “use-it-or-lose-it” to stress timely program evaluation use. Yet the literature on program evaluation in GME often neglects to discuss use, including how selected evaluation approaches can influence evaluation use.1,6,7 In this article, we explain evaluation use by describing both the use of evaluation findings and process use (ie, changes resulting from engagement in the evaluation process itself).1,8 We also suggest strategies, including evaluation approaches, that faculty can use to increase evaluation use in GME.The 3 categories of use of evaluation findings are instrumental, conceptual, and symbolic. Instrumental use refers to instances where stakeholders use evaluation findings to take direct actions (eg, improvements, changes, terminations) in a program.9 For example, evaluation findings show that residents in a GME program are struggling to complete their research projects. Using the findings, the GME team implements new research training activities to assist residents in the completion of their projects. Conceptual use describes occurrences where stakeholders use evaluation findings to evolve their understandings of a program but do not take direct actions based on these findings.4 For instance, the GME team acknowledges the findings that residents are struggling to complete their research projects. These findings inform their understanding of why residents are not attending academic conferences to present their research. Lastly, symbolic use occurs when stakeholders use the sheer existence of a completed evaluation to comply with reporting requirements or justify a previously made program action.4 For example, the funding university requires the GME program to complete an evaluation to retain funding for residents' research projects. The GME team completes an evaluation and presents the report to the university. Alternatively, before the evaluation, the GME program hired a research assistant to help residents with their research projects and the subsequent evaluation findings are used to justify the hiring of the research assistant. In GME, we emphasize instrumental use, as this form of use leads to actions that can improve programs. However, the use of evaluation findings is typically a short-term consequence of evaluation because these findings are relevant only within a specific and limited timeframe (ie, use-it-or-lose it).On the other hand, process use can have ongoing influence on individuals, programs, and organizations. It recognizes that evaluation processes themselves can affect attitudes, thought processes, and behaviors.10 Process use recognizes stakeholders' learning advancements from their involvement in an evaluation as well as the effects of evaluation processes on program functioning and organizational culture.11 Process use does not require changes to a program or direct actions because of evaluation findings. There are 6 types of process use which we illustrate with examples:When stakeholders are involved in evaluation processes, they enter an evaluation culture and learn how to think and look at things through an evaluative lens. They can also use the knowledge and skills (eg, evaluation knowledge, methodological and facilitation skills) they develop to strengthen their organization's abilities to design, implement, interpret, and use evaluations and thereby build their organization's evaluation capacity. In this sense, process use is valuable throughout and following an evaluation and in various GME settings regardless of the evaluation findings or recommendations.12The Table presents strategies that faculty involved in program evaluation can employ to increase evaluation use.In closing, it is imperative to remember that evaluation use, especially process use, can occur throughout a program evaluation rather than simply at its conclusion.10 Evaluation use can start at the planning stage and continue well beyond a presentation or publication of an evaluation. Program evaluators need a use-it-or-lose-it perspective throughout the evaluation process to maximize improvements to training. This perspective will maintain stakeholders' faith in the value of evaluation, as they witness that evaluation efforts lead to timely, actionable findings and processes. Ultimately, we must embrace evaluation use to ensure that all stakeholders and programs, not only conference attendees, readers of peer-reviewed journals, or our CVs, witness the consequences (both positive and negative) of program evaluation.