Sylvia Heeneman
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Research Interests
Sylvia Heeneman has a total of 141 co-authors. Here are the top co-authors by number of shared publications:
- Cees P. M. van der Vleuten
Department of Educational Development and Research in the Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands
5 shared publications - Erik W. Driessen
Maastricht University
4 shared publications - Mat J. A. P. Daemen
Department of Pathology, AMC, Amsterdam, The Netherlands
4 shared publications - Suzanne Schut
Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Maastricht University, Maastricht, the Netherlands
4 shared publications - Dario Torre
Department of Medical Education University of Torino School of Medicine Turin Italy
3 shared publications - Jamiu O. Busari
Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands
3 shared publications
Latest Publications
- Open Access
- 2 October 2026
- Published by Wiley in Medical Education
The whole is greater than the sum of its parts: Evaluating systems of assessment as complex adaptive systems
Background: Assessment in health professions education has evolved from individual assessment instruments towards integrated systems of assessment that support learning and high‐stakes decision‐making. While considerable attention has been devoted to the design, validity and reliability of individual assessment methods, far less attention has been paid to evaluating assessment systems themselves. This is an important gap, as assessment systems may contain multiple high‐quality assessment tools yet still fail to function as coherent and effective wholes. Purpose: This paper proposes a conceptual framework for evaluating systems of assessment in health professions education. Drawing on systems thinking and complexity theory, we argue that contemporary assessment systems, such as programmatic assessment, can be understood as complex adaptive systems and therefore require evaluation approaches that extend beyond the evaluation of individual assessment components. Approach: We trace the evolution of assessment thinking—from individual assessments to systems of assessment—and consider these systems through the lens of complex adaptive systems. We then combine the Logic Model and the Context–Input–Process–Product (CIPP) evaluation framework to connect programme theory with evaluation and examine both intended and unintended consequences. Framework: The integrated framework comprises six components: Planning, Resource Design, Implementation, Short‐term Results, Outcomes and Long‐term Impact. Together, these components provide a structure for evaluating how assessment systems are designed, implemented, experienced and improved over time. A worked example illustrates how system‐level evaluation identifies issues that are not apparent when individual assessment tools are evaluated in isolation. Conclusions: Evaluating assessment systems requires approaches that examine interactions, implementation processes, educational impact and broader consequences. The proposed framework offers a structured theory‐informed approach to support continuous improvement, quality assurance and the development of validity arguments for contemporary systems of assessment. Future research should explore the framework's feasibility, utility and impact across diverse health professions education contexts.
- 2 October 2026
- Published by Wiley in Medical Education
Open AccessThe whole is greater than the sum of its parts: Evaluating systems of assessment as complex adaptive systems
Background: Assessment in health professions education has evolved from individual assessment instruments towards integrated systems of assessment that support learning and high‐stakes decision‐making. While considerable attention has been devoted to the design, validity and reliability of individual assessment methods, far less attention has been paid to evaluating assessment systems themselves. This is an important gap, as assessment systems may contain multiple high‐quality assessment tools yet still fail to function as coherent and effective wholes. Purpose: This paper proposes a conceptual framework for evaluating systems of assessment in health professions education. Drawing on systems thinking and complexity theory, we argue that contemporary assessment systems, such as programmatic assessment, can be understood as complex adaptive systems and therefore require evaluation approaches that extend beyond the evaluation of individual assessment components. Approach: We trace the evolution of assessment thinking—from individual assessments to systems of assessment—and consider these systems through the lens of complex adaptive systems. We then combine the Logic Model and the Context–Input–Process–Product (CIPP) evaluation framework to connect programme theory with evaluation and examine both intended and unintended consequences. Framework: The integrated framework comprises six components: Planning, Resource Design, Implementation, Short‐term Results, Outcomes and Long‐term Impact. Together, these components provide a structure for evaluating how assessment systems are designed, implemented, experienced and improved over time. A worked example illustrates how system‐level evaluation identifies issues that are not apparent when individual assessment tools are evaluated in isolation. Conclusions: Evaluating assessment systems requires approaches that examine interactions, implementation processes, educational impact and broader consequences. The proposed framework offers a structured theory‐informed approach to support continuous improvement, quality assurance and the development of validity arguments for contemporary systems of assessment. Future research should explore the framework's feasibility, utility and impact across diverse health professions education contexts.
- Open Access
- 26 August 2026
- Published by Springer Nature in Journal of Formative Design in Learning
- p. 1-15
Usefulness and Application of a Co-Designed Formative Assessment Practice Perceived by Teachers in Primary Education
Formative assessment practices developed in co-design by teachers have the potential to support novel teachers with its implementation. This study examines the perceived usefulness and application of a co-designed formative assessment practice by primary school teachers that were not involved in the co-design process itself. A convergent mixed-method approach was used that integrated both qualitative and quantitative methods. A qualitative descriptive design was used to provide a comprehensive understanding of the program’s usefulness as perceived by teachers with or without additional training. A quantitative method was used to evaluate the extent to which formative assessment was applied across both groups. The quantitative data was situated within the qualitative framework for the interpretation of findings. Findings indicate that (i) a co-designed formative assessment practices program supports teachers with the implementation of formative assessment to their local teaching contexts. (ii) A comprehensive adoption, unlike procedural approaches to formative assessment, requires additional support in the form of a complementary teacher professional development program. However, this additional support requires far less time than the time investment put into the co-design process itself.
- 26 August 2026
- Published by Springer Nature in Journal of Formative Design in Learning
- p. 1-15
Open AccessUsefulness and Application of a Co-Designed Formative Assessment Practice Perceived by Teachers in Primary Education
Formative assessment practices developed in co-design by teachers have the potential to support novel teachers with its implementation. This study examines the perceived usefulness and application of a co-designed formative assessment practice by primary school teachers that were not involved in the co-design process itself. A convergent mixed-method approach was used that integrated both qualitative and quantitative methods. A qualitative descriptive design was used to provide a comprehensive understanding of the program’s usefulness as perceived by teachers with or without additional training. A quantitative method was used to evaluate the extent to which formative assessment was applied across both groups. The quantitative data was situated within the qualitative framework for the interpretation of findings. Findings indicate that (i) a co-designed formative assessment practices program supports teachers with the implementation of formative assessment to their local teaching contexts. (ii) A comprehensive adoption, unlike procedural approaches to formative assessment, requires additional support in the form of a complementary teacher professional development program. However, this additional support requires far less time than the time investment put into the co-design process itself.
- Open Access
- 16 June 2026
- Published by JMIR Publications Inc. in JMIR Medical Education
- Vol. 12 (1) , e90064
AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study
Background: Feedback is essential for medical students’ learning during clinical clerkships; yet, supervising physicians often struggle to provide meaningful written feedback due to time constraints. Large language models offer a promising approach to supplement human feedback, but how artificial intelligence (AI)–generated and human feedback differ in authentic clinical settings remains unclear, as most comparisons have been conducted in classroom or simulation contexts. Objective: The aim of the study is to examine how AI-generated feedback and supervisor-provided feedback differ when applied to medical students’ clinical clerkship logs, by identifying the distinct characteristics and complementary strengths of each feedback type. Methods: This cross-sectional convergent mixed methods study included 161 weekly clinical clerkship logs from 47 fifth- and sixth-year medical students across 12 clinical departments at Nagoya University, Japan (January-May 2024). Of 164 eligible logs, 3 were excluded because supervisors entered contact messages rather than substantive feedback. AI feedback was generated using GPT-4o. In total, 10 faculty physicians and 10 medical students evaluated both feedback types in blinded, randomized order using a validated 5-category rubric (criteria-based, clear direction, accuracy, prioritization, and supportive tone), followed by open-ended comments and source identification. Quantitative analyses (paired 2-tailed t tests, cumulative link mixed-effects models; α=.05 with Bonferroni correction) were complemented by qualitative thematic analysis and integrated using joint display analysis. Results: AI feedback was significantly longer than supervisor feedback (mean 382.02, SD 81.82 vs mean 98.87, SD 73.66 characters; Cohen d =2.84, 95% CI 2.50‐3.19; P <.001). Cumulative link mixed-effects models showed that AI scored higher on criteria-based (odds ratio [OR] 11.81, 95% CI 7.64‐18.27; P <.001) and clear direction (OR 6.61, 95% CI 4.35‐10.06; P <.001), with no significant differences on accuracy (OR 1.35, 95% CI 0.91‐2.00; P >.99), prioritization (OR 1.70, 95% CI 1.16‐2.50; P =.10), or supportive tone (OR 1.34, 95% CI 0.87‐2.06; P >.99). AI feedback showed greater consistency (variance ratio 3.9:1; Levene F 1,320 =73.20; P <.001). All 20 evaluators correctly identified feedback sources. Qualitative analysis revealed that AI provided structured, text-anchored feedback addressing rubric criteria, while supervisors offered experience-based feedback grounded in clinical context and professional expertise. Conclusions: This study extends the comparison of AI-generated and supervisor feedback to an authentic clinical clerkship environment, moving beyond classroom and simulation settings examined in prior work. Through integrated mixed methods analysis, a key distinction emerged between text-anchored AI feedback, which systematically addresses written log content in alignment with rubric criteria, and experience-based supervisor feedback, which draws on clinical observation and professional judgment. AI consistently delivered structured feedback addressing gaps that arise when time-pressured supervisors provide brief comments, while supervisors contributed clinically grounded insights that AI cannot replicate. These complementary strengths suggest that AI feedback should supplement rather than replace supervisor feedback, and that hybrid models leveraging each type’s advantages warrant investigation in clinical education.
- 16 June 2026
- Published by JMIR Publications Inc. in JMIR Medical Education
- Vol. 12 (1) , e90064
Open AccessAI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study
Background: Feedback is essential for medical students’ learning during clinical clerkships; yet, supervising physicians often struggle to provide meaningful written feedback due to time constraints. Large language models offer a promising approach to supplement human feedback, but how artificial intelligence (AI)–generated and human feedback differ in authentic clinical settings remains unclear, as most comparisons have been conducted in classroom or simulation contexts. Objective: The aim of the study is to examine how AI-generated feedback and supervisor-provided feedback differ when applied to medical students’ clinical clerkship logs, by identifying the distinct characteristics and complementary strengths of each feedback type. Methods: This cross-sectional convergent mixed methods study included 161 weekly clinical clerkship logs from 47 fifth- and sixth-year medical students across 12 clinical departments at Nagoya University, Japan (January-May 2024). Of 164 eligible logs, 3 were excluded because supervisors entered contact messages rather than substantive feedback. AI feedback was generated using GPT-4o. In total, 10 faculty physicians and 10 medical students evaluated both feedback types in blinded, randomized order using a validated 5-category rubric (criteria-based, clear direction, accuracy, prioritization, and supportive tone), followed by open-ended comments and source identification. Quantitative analyses (paired 2-tailed t tests, cumulative link mixed-effects models; α=.05 with Bonferroni correction) were complemented by qualitative thematic analysis and integrated using joint display analysis. Results: AI feedback was significantly longer than supervisor feedback (mean 382.02, SD 81.82 vs mean 98.87, SD 73.66 characters; Cohen d =2.84, 95% CI 2.50‐3.19; P <.001). Cumulative link mixed-effects models showed that AI scored higher on criteria-based (odds ratio [OR] 11.81, 95% CI 7.64‐18.27; P <.001) and clear direction (OR 6.61, 95% CI 4.35‐10.06; P <.001), with no significant differences on accuracy (OR 1.35, 95% CI 0.91‐2.00; P >.99), prioritization (OR 1.70, 95% CI 1.16‐2.50; P =.10), or supportive tone (OR 1.34, 95% CI 0.87‐2.06; P >.99). AI feedback showed greater consistency (variance ratio 3.9:1; Levene F 1,320 =73.20; P <.001). All 20 evaluators correctly identified feedback sources. Qualitative analysis revealed that AI provided structured, text-anchored feedback addressing rubric criteria, while supervisors offered experience-based feedback grounded in clinical context and professional expertise. Conclusions: This study extends the comparison of AI-generated and supervisor feedback to an authentic clinical clerkship environment, moving beyond classroom and simulation settings examined in prior work. Through integrated mixed methods analysis, a key distinction emerged between text-anchored AI feedback, which systematically addresses written log content in alignment with rubric criteria, and experience-based supervisor feedback, which draws on clinical observation and professional judgment. AI consistently delivered structured feedback addressing gaps that arise when time-pressured supervisors provide brief comments, while supervisors contributed clinically grounded insights that AI cannot replicate. These complementary strengths suggest that AI feedback should supplement rather than replace supervisor feedback, and that hybrid models leveraging each type’s advantages warrant investigation in clinical education.
- Open Access
- 13 June 2026
- Published by Taylor & Francis in Medical Teacher
- p. 1-10
Feedback focus: An exploration of documented feedback conversations with episodic versus continuous supervisors
Feedback conversations are essential to learning, and the relationship between supervisor and learner plays a role in the effectiveness of feedback shared. Documentation of these conversations in learner portfolios is intended to support reflection, learning, and growth. However, there is limited understanding of what gets documented after feedback conversations. Guided by Hattie and Timperley's model for feedback, we examined documented feedback conversations with episodic and continuous supervisors to better understand the information contained in learner portfolios. The authors examined archived de-identified FieldNotes (FNs; workplace-based assessment forms) completed for residents located at two teaching sites in a Canadian university-based family medicine residency program. The content of written feedback on each FN was coded for feedback focus (Task, Process, Self-regulation and Self) from Hattie and Timperley's model, and proportions of foci were compared between episodic supervisors (ES) and continuous supervisors (CS). A total of 10,624 FNs were collected from 158 residents who began and completed their residency training between 2015-2023. The dataset included 5,308 FNs categorized as ES (50%) and 5,316 FNs categorized as CS (50%). A total of 1,332 FNs (12.5%) contained no feedback leaving 9,292 FNs for analysis. A total of 12,960 feedback foci were coded with most feedback focused on Task for all supervisors. CS included Process feedback in a higher proportion than ES. ES included Self feedback in a higher proportion than CS. There were no differences for Task or Self-regulation feedback. This study provides insight into the content of written summaries of feedback conversations included in learner portfolios. There are differences in the content of feedback documented from ES compared with CS. This information can be used to improve the delivery and use of feedback content from different relationships important for learning, reflection and the development of self-regulated learning skills.
- 13 June 2026
- Published by Taylor & Francis in Medical Teacher
- p. 1-10
Open AccessFeedback focus: An exploration of documented feedback conversations with episodic versus continuous supervisors
Feedback conversations are essential to learning, and the relationship between supervisor and learner plays a role in the effectiveness of feedback shared. Documentation of these conversations in learner portfolios is intended to support reflection, learning, and growth. However, there is limited understanding of what gets documented after feedback conversations. Guided by Hattie and Timperley's model for feedback, we examined documented feedback conversations with episodic and continuous supervisors to better understand the information contained in learner portfolios. The authors examined archived de-identified FieldNotes (FNs; workplace-based assessment forms) completed for residents located at two teaching sites in a Canadian university-based family medicine residency program. The content of written feedback on each FN was coded for feedback focus (Task, Process, Self-regulation and Self) from Hattie and Timperley's model, and proportions of foci were compared between episodic supervisors (ES) and continuous supervisors (CS). A total of 10,624 FNs were collected from 158 residents who began and completed their residency training between 2015-2023. The dataset included 5,308 FNs categorized as ES (50%) and 5,316 FNs categorized as CS (50%). A total of 1,332 FNs (12.5%) contained no feedback leaving 9,292 FNs for analysis. A total of 12,960 feedback foci were coded with most feedback focused on Task for all supervisors. CS included Process feedback in a higher proportion than ES. ES included Self feedback in a higher proportion than CS. There were no differences for Task or Self-regulation feedback. This study provides insight into the content of written summaries of feedback conversations included in learner portfolios. There are differences in the content of feedback documented from ES compared with CS. This information can be used to improve the delivery and use of feedback content from different relationships important for learning, reflection and the development of self-regulated learning skills.
- 9 June 2026
- Published by Maastricht University
Learning Cycle Toolbox
The Learning Cycle Toolbox supports students and teachers in self‑regulated learning through a set of concise, practical tools aligned with every phase of the learning cycle. Originally developed for Maastricht University’s medical curriculum, it now serves as an easy‑to‑use resource for any SRL-promoting higher‑education context.
- 9 June 2026
- Published by Maastricht University
Learning Cycle Toolbox
The Learning Cycle Toolbox supports students and teachers in self‑regulated learning through a set of concise, practical tools aligned with every phase of the learning cycle. Originally developed for Maastricht University’s medical curriculum, it now serves as an easy‑to‑use resource for any SRL-promoting higher‑education context.
- Open Access
- 23 May 2026
- Published by Taylor & Francis in Medical Teacher
- Vol. 48 (10) , 1-13
When using entrustable professional activities thwarts basic psychological needs: A case study of residents' experience
Practice pointsPrograms should prioritize educational quality over documentation quantity in EPA implementation.Electronic documentation systems should be designed to facilitate rather than replace...
- 23 May 2026
- Published by Taylor & Francis in Medical Teacher
- Vol. 48 (10) , 1-13
Open AccessWhen using entrustable professional activities thwarts basic psychological needs: A case study of residents' experience
Practice pointsPrograms should prioritize educational quality over documentation quantity in EPA implementation.Electronic documentation systems should be designed to facilitate rather than replace...
- Open Access
- 5 March 2026
- Published by Taylor & Francis in Medical Teacher
- Vol. 48 (10) , 1-12
Co-creation of a fit-for-purpose Feedback Toolkit for clinical clerkships
Feedback is a powerful educational intervention in clinical education, yet its effectiveness depends on how it is integrated into teaching and learning activities. Previous studies have shown that productive feedback in clinical education relies on sociocultural factors such as a supportive feedback culture, trustworthy relationships, and student agency. Co-creation is a promising approach for designing educational interventions that are contextually relevant and aligned with the needs of teachers and students. This study aimed to advance both theoretical and practical understanding of co-creation as a design strategy in health professions education, particularly in developing productive feedback processes tailored to undergraduate clinical education. Eight co-creation sessions were conducted with faculty, clinical teachers, students, and researchers. The process was iterative and grounded in feedback design principles informed by the literature. Co-creation led to the development of a prototype of a Feedback Toolkit, which was piloted in two dyads of clinical-teacher students in a seven-week physiotherapy clerkship. Weekly audio diaries were collected from participants and analyzed using content analysis. Data from the co-creation sessions informed the development of a Feedback Toolkit specifically designed for the clinical teacher-student dyad. The toolkit was built upon three design principles: (1) Contributes to a trustful relationship based on continuous mutual support, (2) Envisioned learning opportunities and feedback scaffolding, and (3) Plan the use of feedback. To operationalize these principles, the toolkit included practical materials such as podcasts, infographics, feedback prompts, and a Mini-CEX. The pilot study demonstrated the toolkit's usability and acceptability and highlighted its value in structuring feedback interactions. Challenges included limited time for full implementation and difficulties in providing constructive feedback. The co-creation approach enabled the development of a fit-for-purpose feedback toolkit that aligns with the dynamic needs of clinical education. This study highlights co-creation as a feasible strategy for designing feedback processes in workplace-based learning.
- 5 March 2026
- Published by Taylor & Francis in Medical Teacher
- Vol. 48 (10) , 1-12
Open AccessCo-creation of a fit-for-purpose Feedback Toolkit for clinical clerkships
Feedback is a powerful educational intervention in clinical education, yet its effectiveness depends on how it is integrated into teaching and learning activities. Previous studies have shown that productive feedback in clinical education relies on sociocultural factors such as a supportive feedback culture, trustworthy relationships, and student agency. Co-creation is a promising approach for designing educational interventions that are contextually relevant and aligned with the needs of teachers and students. This study aimed to advance both theoretical and practical understanding of co-creation as a design strategy in health professions education, particularly in developing productive feedback processes tailored to undergraduate clinical education. Eight co-creation sessions were conducted with faculty, clinical teachers, students, and researchers. The process was iterative and grounded in feedback design principles informed by the literature. Co-creation led to the development of a prototype of a Feedback Toolkit, which was piloted in two dyads of clinical-teacher students in a seven-week physiotherapy clerkship. Weekly audio diaries were collected from participants and analyzed using content analysis. Data from the co-creation sessions informed the development of a Feedback Toolkit specifically designed for the clinical teacher-student dyad. The toolkit was built upon three design principles: (1) Contributes to a trustful relationship based on continuous mutual support, (2) Envisioned learning opportunities and feedback scaffolding, and (3) Plan the use of feedback. To operationalize these principles, the toolkit included practical materials such as podcasts, infographics, feedback prompts, and a Mini-CEX. The pilot study demonstrated the toolkit's usability and acceptability and highlighted its value in structuring feedback interactions. Challenges included limited time for full implementation and difficulties in providing constructive feedback. The co-creation approach enabled the development of a fit-for-purpose feedback toolkit that aligns with the dynamic needs of clinical education. This study highlights co-creation as a feasible strategy for designing feedback processes in workplace-based learning.
- Open Access
- 24 December 2025
- Published by JMIR Publications Inc. in JMIR Preprints
Same logs, different voices: AI-generated vs human feedback during clinical clerkship in undergraduate education (Preprint)
BACKGROUND Feedback is essential for medical students' learning during clinical clerkships, yet supervising physicians often struggle to provide meaningful written feedback due to time constraints. Large language models (LLMs) offer a promising approach to supplement human feedback, but how AI-generated and human feedback differ in authentic clinical settings remains unclear. Previous studies have yielded inconsistent findings regarding feedback length, quality, and distinguishability, with most comparisons conducted in classroom or simulation contexts rather than clinical environments. OBJECTIVE To examine how AI-generated feedback and supervisor-provided feedback differ when applied to medical students' clinical clerkship logs, by identifying the distinct characteristics and complementary strengths of each feedback type. METHODS This mixed-methods study employed a convergent design. We collected 161 sets of weekly clinical clerkship logs from fifth- and sixth-year medical students at Nagoya University, Japan, along with corresponding supervisor feedback and AI-generated feedback using GPT-4o. Ten faculty members and ten medical students evaluated both feedback types using a validated rubric assessing five categories: criteria-based, clear directions for improvement, accuracy, prioritization, and supportive tone. Quantitative analyses included paired t-tests, cumulative link mixed models, and correlation analyses. Qualitative thematic analysis examined evaluators' open-ended comments. Results were integrated using Joint Display Analysis. RESULTS AI feedback was significantly longer than supervisor feedback (mean: 382 vs. 98 characters, p <0.001). AI feedback scored significantly higher on criteria-based (OR=11.81, p<0.001) and clear direction (OR=6.61, p<0.001) categories, with no significant differences in accuracy, prioritization, or supportive tone. AI feedback demonstrated greater quality consistency, while supervisor feedback showed higher variability (variance ratio 3.9:1). For supervisor feedback, length positively correlated with quality scores; no such correlation existed for AI feedback. All evaluators correctly identified feedback sources. Qualitative analysis of open-ended comments revealed five themes: adherence to feedback criteria and structure, continuity and consistency, perspective as a clinician, quality of Japanese language, and text length. AI provided structured, text-anchored feedback following rubric criteria, while supervisors offered experience-based feedback grounded in clinical context and professional expertise that sometimes lacked structured elements. CONCLUSIONS AI-generated and supervisor-provided feedback show distinct but complementary strengths. AI consistently delivers structured, criterion-based feedback aligned with written content, addressing gaps that may arise when time-pressured supervisors provide brief feedback. However, AI lacks the clinical perspective and contextual grounding that supervisors bring from direct observation and professional experience. These findings suggest that AI feedback should complement rather than replace human feedback in clinical clerkship settings, with each type addressing the other's limitations to optimize student learning.
- 24 December 2025
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessSame logs, different voices: AI-generated vs human feedback during clinical clerkship in undergraduate education (Preprint)
BACKGROUND Feedback is essential for medical students' learning during clinical clerkships, yet supervising physicians often struggle to provide meaningful written feedback due to time constraints. Large language models (LLMs) offer a promising approach to supplement human feedback, but how AI-generated and human feedback differ in authentic clinical settings remains unclear. Previous studies have yielded inconsistent findings regarding feedback length, quality, and distinguishability, with most comparisons conducted in classroom or simulation contexts rather than clinical environments. OBJECTIVE To examine how AI-generated feedback and supervisor-provided feedback differ when applied to medical students' clinical clerkship logs, by identifying the distinct characteristics and complementary strengths of each feedback type. METHODS This mixed-methods study employed a convergent design. We collected 161 sets of weekly clinical clerkship logs from fifth- and sixth-year medical students at Nagoya University, Japan, along with corresponding supervisor feedback and AI-generated feedback using GPT-4o. Ten faculty members and ten medical students evaluated both feedback types using a validated rubric assessing five categories: criteria-based, clear directions for improvement, accuracy, prioritization, and supportive tone. Quantitative analyses included paired t-tests, cumulative link mixed models, and correlation analyses. Qualitative thematic analysis examined evaluators' open-ended comments. Results were integrated using Joint Display Analysis. RESULTS AI feedback was significantly longer than supervisor feedback (mean: 382 vs. 98 characters, p <0.001). AI feedback scored significantly higher on criteria-based (OR=11.81, p<0.001) and clear direction (OR=6.61, p<0.001) categories, with no significant differences in accuracy, prioritization, or supportive tone. AI feedback demonstrated greater quality consistency, while supervisor feedback showed higher variability (variance ratio 3.9:1). For supervisor feedback, length positively correlated with quality scores; no such correlation existed for AI feedback. All evaluators correctly identified feedback sources. Qualitative analysis of open-ended comments revealed five themes: adherence to feedback criteria and structure, continuity and consistency, perspective as a clinician, quality of Japanese language, and text length. AI provided structured, text-anchored feedback following rubric criteria, while supervisors offered experience-based feedback grounded in clinical context and professional expertise that sometimes lacked structured elements. CONCLUSIONS AI-generated and supervisor-provided feedback show distinct but complementary strengths. AI consistently delivers structured, criterion-based feedback aligned with written content, addressing gaps that may arise when time-pressured supervisors provide brief feedback. However, AI lacks the clinical perspective and contextual grounding that supervisors bring from direct observation and professional experience. These findings suggest that AI feedback should complement rather than replace human feedback in clinical clerkship settings, with each type addressing the other's limitations to optimize student learning.
- Open Access
- 14 August 2025
- Published by Wiley in Medical Education
- Vol. 60 (3) , 307-316
The transformative power of a change laboratory: Experiences from an assessment re‐design project in Oman
Introduction: Research on educational change consistently shows that the success of any implementation process depends on the involvement of stakeholders. The Change Laboratory (CL) is a method that involves stakeholders in co‐creating their work practice. Its application in medical education has been proven to result in improvement of learning environments. The aim of this study is to explore participants' perceptions on how the use of the CL impacts their commitment to change. Methods: We conducted a qualitative study to explore participants' experiences with a CL intervention focused on redesigning a workplace‐based assessment program in postgraduate medical training in Oman. Semi‐structured interviews with nine participants (residents, faculty and administrators) were conducted to examine how the CL process influenced their commitment to change, focusing on learning and agency. Data collection and analysis followed an iterative process. We used a reflexive thematic analysis approach and constructed a model of how the CL intervention resulted in transformative changes at the individual, team and organization levels. Results: Participants' intrinsic motivation to actively engage in assessment re‐design was driven by perceptions of a shared understanding of tensions in the assessment system and the realization that, as a group, they would be able to initiate meaningful change. Three shifts experienced by participants while progressing through the CL were identified: from individual learning to collective competence; from individual responsibility to collective agency; and, from collective agency to organizational learning, reflecting engagement in change processes beyond participants' residency training program. Conclusion: By illustrating how a CL intervention fosters transformative shifts in learning and agency, at individual, collective and organizational levels, this study extends prior work on participatory design in medical education. It provides empirical evidence that co‐creation supports both personal and professional development, fostering stakeholders' competencies to collaboratively navigate and lead educational change processes.
- 14 August 2025
- Published by Wiley in Medical Education
- Vol. 60 (3) , 307-316
Open AccessThe transformative power of a change laboratory: Experiences from an assessment re‐design project in Oman
Introduction: Research on educational change consistently shows that the success of any implementation process depends on the involvement of stakeholders. The Change Laboratory (CL) is a method that involves stakeholders in co‐creating their work practice. Its application in medical education has been proven to result in improvement of learning environments. The aim of this study is to explore participants' perceptions on how the use of the CL impacts their commitment to change. Methods: We conducted a qualitative study to explore participants' experiences with a CL intervention focused on redesigning a workplace‐based assessment program in postgraduate medical training in Oman. Semi‐structured interviews with nine participants (residents, faculty and administrators) were conducted to examine how the CL process influenced their commitment to change, focusing on learning and agency. Data collection and analysis followed an iterative process. We used a reflexive thematic analysis approach and constructed a model of how the CL intervention resulted in transformative changes at the individual, team and organization levels. Results: Participants' intrinsic motivation to actively engage in assessment re‐design was driven by perceptions of a shared understanding of tensions in the assessment system and the realization that, as a group, they would be able to initiate meaningful change. Three shifts experienced by participants while progressing through the CL were identified: from individual learning to collective competence; from individual responsibility to collective agency; and, from collective agency to organizational learning, reflecting engagement in change processes beyond participants' residency training program. Conclusion: By illustrating how a CL intervention fosters transformative shifts in learning and agency, at individual, collective and organizational levels, this study extends prior work on participatory design in medical education. It provides empirical evidence that co‐creation supports both personal and professional development, fostering stakeholders' competencies to collaboratively navigate and lead educational change processes.
- Open Access
- 28 February 2025
- Published by Association for Learning Development in Higher Education in Journal of Learning Development in Higher Education
- No. 34
Capturing meaningful moments: a narrative analysis of selfies in medical student portfolios
Portfolios are designed to promote self-regulated learning and reflective practice through guided reflection on collections of artefacts. However, many medical students are reluctant to write reflections. On social media and in social research, photographic self-portraits (selfies) document experiences, relationships, and identity. Therefore, selfies may present an unexplored alternative to written reflections. We explored how selfies in student portfolios might support reflection on learning and professional identity development (PID) during the first two years of medical school. Our longitudinal qualitative study analysed 200 selfies, including individual and group photos or videos created by 37 students over 14 months between May 2020 and July 2021. These selfies were included in student presentations alongside other artefacts for interviews with portfolio advisors. The student sample was purposefully selected from a cohort of 147 first-year medical students. Visual narrative analysis identified recurring stories about the participants’ clinical and social learning. Selfies documented classwork, social activities, and clinical learning associated with stories related to Beginning, Connection, Shared Activities, and Belonging. Viewed in combination, the stories revealed narratives of Integration, Competence, and PID consistent with the stages described in situated learning theory. Selfies can offer an alternative or addition to written reflections for engaging and supporting medical students’ reflections on some aspects of their learning experiences. Selfies are relevant to team-based and clinical learning and PID, complementing evidence of academic achievements. Ethical guidelines for their use and further studies on their use in other educational contexts are needed.
- 28 February 2025
- Published by Association for Learning Development in Higher Education in Journal of Learning Development in Higher Education
- No. 34
Open AccessCapturing meaningful moments: a narrative analysis of selfies in medical student portfolios
Portfolios are designed to promote self-regulated learning and reflective practice through guided reflection on collections of artefacts. However, many medical students are reluctant to write reflections. On social media and in social research, photographic self-portraits (selfies) document experiences, relationships, and identity. Therefore, selfies may present an unexplored alternative to written reflections. We explored how selfies in student portfolios might support reflection on learning and professional identity development (PID) during the first two years of medical school. Our longitudinal qualitative study analysed 200 selfies, including individual and group photos or videos created by 37 students over 14 months between May 2020 and July 2021. These selfies were included in student presentations alongside other artefacts for interviews with portfolio advisors. The student sample was purposefully selected from a cohort of 147 first-year medical students. Visual narrative analysis identified recurring stories about the participants’ clinical and social learning. Selfies documented classwork, social activities, and clinical learning associated with stories related to Beginning, Connection, Shared Activities, and Belonging. Viewed in combination, the stories revealed narratives of Integration, Competence, and PID consistent with the stages described in situated learning theory. Selfies can offer an alternative or addition to written reflections for engaging and supporting medical students’ reflections on some aspects of their learning experiences. Selfies are relevant to team-based and clinical learning and PID, complementing evidence of academic achievements. Ethical guidelines for their use and further studies on their use in other educational contexts are needed.