Ylva Holzhausen

Background: Clinical summary statements convert unstructured patient information into a concise format. They are widely used in virtual patients (VPs) to train medical students in diagnostic reasoning. Most VP systems provide little individualized feedback on the quality of learners’ summaries, and earlier automated approaches have required complex natural language processing, limiting their scalability. Large language models (LLMs) may offer a more flexible, rubric-guided way to score summary statements and support feedback. Still, their test-retest reliability and concordance with expert ratings in this context remain unclear. Objective: This study aims to evaluate the test-retest reliability—specifically, whether the models produce stable ratings across repetitions—of LLM-generated medical summary statement ratings, and to measure their concordance with human expert assessments across different prompting strategies. Methods: We conducted an experimental comparative study using 122 rubric-scored summary statements (English and German) from 6 internal medicine VP cases previously rated by 2 expert raters. Statements were evaluated along 4 ternary components (semantic qualifiers, transformation, narrowing, and global rating), and 2 binary components (factual accuracy and person). Four LLMs (Anthropic’s Claude, OpenAI’s GPT-4o, Mistral AI’s Mistral Large, and Mistral AI’s Mistral 7B) were each run 5 times under 4 prompting strategies (zero-shot, zero-shot with chain-of-thought [CoT], few-shot, and few-shot+CoT). The test-retest reliability was assessed with intraclass correlation coefficients (ICCs), Fleiss kappa (κ), and concordance with human ratings with accuracy (%) and chance-corrected interrater agreement (Gwet AC1 and AC2). Results: Large closed-weights models (Claude, GPT-4o, and Mistral Large) showed high test-retest reliability (ICC≥0.92; Fleiss κ=0.59‐1.00). Claude was essentially deterministic (ICC and Fleiss κ≥0.99); GPT-4o and Mistral Large reached ICC≥0.97 and Fleiss κ≥0.80 except under zero-shot. Mistral 7B, a small open-weights model, had high ICCs (≥0.88) but low Fleiss κ (0.32‐0.52). Few-shot+CoT prompting achieved the best concordance with human ratings for all the models (means across runs and rubric components), peaking for Mistral Large (accuracy 77%; Gwet AC1 and AC2 0.85, 95% CI 0.83‐0.87), followed by Claude (71%; 0.77, 0.74‐0.81), GPT-4o (67%; 0.72, 0.68‐0.77), and Mistral 7B (60%; 0.64, 0.56‐0.73). Concordance was higher for the binary rubric components (person and factual accuracy) than for the ternary components. Extreme deviations and invalid outputs were rare but more frequent in ternary components and the smaller local model. Conclusions: Large, general-purpose LLMs can function as reliable, rubric-guided raters of VP-based clinical summary statements, achieving high concordance with expert ratings under clearly defined criteria in binary rubric components and when using few-shot with CoT prompting. The performance on more nuanced, ternary reasoning dimensions was more variable. LLMs can support scalable, rubric-based formative feedback on clinical reasoning within VP systems but should complement rather than replace human judgment.

BACKGROUND Clinical summary statements convert unstructured patient information into a concise format. They are widely used in virtual patients (VPs) to train medical students in diagnostic reasoning. Most VP systems provide little individualized feedback on the quality of learners’ summaries, and earlier automated approaches have required complex natural language processing, limiting their scalability. Large language models (LLMs) may offer a more flexible, rubric-guided way to score summary statements and support feedback, but their test–retest reliability and concordance with expert ratings in this context remain unclear. OBJECTIVE To examine the test–retest reliability of LLM-generated medical summary statement ratings as well as the concordance with human expert assessments under different prompting strategies. METHODS We conducted an experimental comparative study using 122 rubric-scored summary statements (English and German) from six internal medicine VP cases previously rated by two expert raters. Statements were evaluated along four ternary components (Semantic Qualifiers, Transformation, Narrowing, Global Rating) and two binary components (Factual Accuracy, Person). Four LLMs (Claude, GPT-4o, Mistral Large, Mistral 7B) were run under four prompting strategies (zero-shot, zero-shot with chain-of-thought [CoT], few-shot, few-shot+CoT), which were repeated five times. The test–retest reliability was assessed with intraclass correlation coefficients (ICCs) and Fleiss’ kappa (κ). The concordance with human ratings was quantified using accuracy (%) and chance-corrected interrater agreement (Gwet’s AC1/AC2). RESULTS All the models showed high test–retest reliability (ICC ≥0.89 and Fleiss’ kappa 0.32–1.00). Claude was essentially deterministic (ICC ≥0.99; Fleiss’ kappa ≥0.99), whereas GPT-4o and Mistral Large also showed high test–retest reliability (ICC ≥0.97; Fleiss’ kappa ≥0.80, excluding zero-shot), whereas Mistral 7B had high ICCs (≥0.88) but relatively low Fleiss’ kappa values (0.32–0.53). Few-shot with CoT prompting achieved the best concordance with human ratings for all the models, peaking for Mistral Large (accuracy: mean=77%; Gwet’s AC1/AC2: mean=0.85, CI=0.83–0.87), followed by Claude (accuracy: mean=71%; Gwet’s AC1/AC2: mean=0.77, CI=0.74–0.81), GPT-4o (accuracy: mean=67%; Gwet’s AC1/AC2: mean=0.72, CI: 0.68–0.77), and Mistral 7B (accuracy: mean=60%; Gwet’s AC1/AC2: mean=0.64, CI: 0.56–0.73). The concordance with human ratings was higher for the binary rubric components (Person, Factual Accuracy) than for the ternary components. Extreme deviations and invalid outputs were rare but more frequent in ternary components and the smaller local model. CONCLUSIONS Large, general-purpose LLMs can function as reliable, rubric-guided raters of VP-based clinical summary statements, achieving high concordance with expert ratings under clearly defined criteria and when using few-shot with CoT prompting. The performance on more nuanced, ternary reasoning dimensions was more variable. LLMs can support scalable, rubric-based formative feedback on clinical reasoning within VP systems but should complement rather than replace human judgment.

Zusammenfassung: Hintergrund: Das chirurgische Tertial im Praktischen Jahr (PJ) soll allen Medizinstudierenden zentrale Elemente der chirurgischen Tätigkeit vermitteln. Wenig ist darüber bekannt, über welche relevanten praktischen Vorerfahrungen die Studierenden vor dem PJ bereits verfügen. Ziel der Arbeit: Ziel dieser Studie ist es, die praktischen Vorerfahrungen von Medizinstudierenden zu Beginn des PJ im chirurgischen Tertial systematisch zu erfassen. Material und Methoden: Die Vorerfahrungen wurden anhand des Konzepts der „Entrustable Professional Activities“ (EPAs) erfasst. PJ-Studierende in der Chirurgie an der Charité und deren Lehrkrankenhäusern dokumentieren ihre Vorerfahrungen in einem verpflichtenden e‑Portfolio, das auf 10 EPAs sowie 20 Prozeduren des NKLM 2.0 basiert. Analysiert wurde, wie eigenständig die EPAs und Prozeduren vor Beginn des ersten Tertials bereits durchgeführt wurden. Ergebnisse: Einbezogen wurden die Angaben von 516 Studierenden zwischen Mai 2023 und Mai 2025. Insgesamt zeigte sich eine große individuelle Variabilität in den praktischen Vorerfahrungen der Studierenden mit den unterschiedlichen EPAs und Prozeduren. Ein relevanter Anteil der Studierenden berichtete bei 7 der 10 EPAs und bei 9 der 20 Prozeduren keine bzw. geringe Vorerfahrungen. Diskussion: Die Studierenden starten mit unterschiedlichen praktischen Vorerfahrungen in das chirurgische PJ-Tertial. Eine strukturierte Erhebung der individuellen Vorerfahrungen auf Basis des EPA-Konzepts ermöglicht es, tätigkeitsbezogene Lernstände konkret sichtbar zu machen, gezielt die Zuweisung von Aufgaben und Supervision zu gestalten und individuelle Lernpläne zu entwickeln. Perspektivisch kann dies zu einer praxisnäheren chirurgischen Lehre im PJ und zu einer besseren praktischen Vorbereitung auf die ärztliche Tätigkeit beitragen und somit auch das Interesse an einer Facharztausbildung in der Chirurgie steigern. Graphic

Entrustable professional activities (EPAs) for entry into residency often remain broad and underspecified and lack alignment with the clinical contexts that medical graduates encounter. This study aims to identify chief complaints that can serve as context anchors for EPAs for entry into residency. A structured, expert consensus survey was conducted in 2024 at Charité-Universitätsmedizin Berlin, Germany. In total, 192 participants from four groups (senior medical students, hospital residents, hospital specialists, and general practitioners) rated 134 chief complaints. An 80% threshold was set to define a consensus on the relevance of complaints for autonomous differential diagnostic work-up by new residents under supervision level 3b. Forty-four chief complaints (33%) reached the consensus threshold, with abdominal pain, dyspnoea, fever and cough receiving the highest agreement (>97%). Strong consistency was observed across expert groups, with all four groups agreeing on 35 complaints. For the remaining nine, only minor variations were observed, with generally only one expert group falling below the 80% threshold. Defining core chief complaints offers a practical approach to contextualize undergraduate EPAs, thereby bridging the gap between educational expectations and real-world clinical practice. These findings support curriculum alignment and entrustment decisions while promoting trainees' readiness for early postgraduate training.

Background: Manual therapy is a fundamental component of physiotherapy education, requiring students to develop complex procedural skills through structured instructional methods. Sweller’s cognitive load theory can provide a framework in manual therapy education on how teaching design affects learning efficiency, especially in the acquisition of procedural skills. This study examines the impact of different teaching approaches on students’ cognitive load in manual therapy education. Methods: A randomized controlled educational study was conducted with 48 physiotherapy students from two cohorts in spring 2022 and 2023. Participants were randomly assigned to an individual practice group (learning one technique at a time) or a series practice group (learning 3–4 techniques simultaneously). A questionnaire assessed global cognitive load as the primary outcome and intrinsic, extraneous, and germane cognitive load as secondary outcomes. Results: 477 questionnaires were analysed. Global cognitive load was significantly lower in the individual group than in the series practice group (MD = -0.55, 95% CI -0.87 to -0.22). Compared to the series practice group, the individual practice group had lower intrinsic cognitive load (MD = -0.29, 95% CI -0.40 to -0.18), slightly lower extraneous cognitive load (MD = -0.07, 95% CI -0.13 to -0.01), and reduced germane cognitive load (MD = -0.29, 95% CI -0.43 to -0.17). Conclusion: Teaching manual therapy one technique at a time may reduce cognitive load, potentially enhancing student learning and performance. This approach underscores the value of applying cognitive load theory in instructional design for physiotherapy education, offering practical benefits for teaching strategies.

Background: The concept of Entrustable Professional Activities (EPAs) is being increasingly adopted for teaching tasks. However, it is debated how entrustment of teaching EPAs compares to entrustment of clinical EPAs. This study investigates the factors influencing the decision to entrust trainees with the supervision of medical students in general practice. Methods: Semi-structured interviews were conducted with nine general practitioners (GPs) and nine GP trainees. The interviews were transcribed and analyzed using the six steps of Braun and Clarke’s thematic analysis. Results: Entrustment of student supervision to GP trainees was influenced by factors related to the GP trainee, the GP, the relationship between the GP trainee and the GP, the teaching task, and the context. Trainee factors included enjoyment of teaching, familiarity with the practice environment, teaching skills and knowledge. Teaching skills were often considered as personality traits rather than learnable skills. Contextual factors included workload and availability of protected teaching time, and availability of students, teachers, and patients. Risks are seen in overburdening trainees and passing on incorrect knowledge. Conclusions: The entrustment of teaching EPAs is influenced by comparable factors as the entrustment of clinical EPAs. Didactic training could support trainees. Explicit entrustment and assessment by direct supervision are not established in the practice context but could help to minimize the risks associated with trainee teaching.

The purpose of this chapter is to conceptualize trust within the context of entrustment in clinical education—to clarify what its purpose is, what its components are, how decisions about it are made, and what other forms of trust it relates to. In a general context, trust is a ubiquitous and intuitive construct that emerges within relationships, enabling individuals to cooperate and collaborate to perform tasks that they might not otherwise be able to perform alone. The trust specific to entrustment emerges from the interdependent goals of patient care and trainee learning, creating reciprocity between supervisor and trainee. Starting from a definition of trust in which risk assessment is central, proposed by Mayer et al. in 1995, additional details are added to conceptualize entrustment’s unique form of trust. Considerations include contrasting the trustworthiness of clinical trainees with that of general trustworthiness, and consolidating the factors that influence entrustment decision-making. The connections between entrustment and other forms of trust within the patient–supervisor–trainee triad are also considered: trainee trust in their supervisors, and patient trust in trainees—including entrustment’s role in ensuring patients’ presumptive trust in trainees is justified. A unified model of entrustment is presented that incorporates these dimensions of trust and their theoretical conceptualizations.

In the evolving landscape of health care education, entrustable professional activities (EPAs), as units of professional practice, provide educators with an authentic means of connecting curricular design with professional practice. This chapter provides health professions educators with a comprehensive and practical guide to EPA development. Navigating the complexities of EPA development demands a nuanced understanding of the underlying constructs. This chapter sequentially tackles fundamental challenges: where to start, identifying key activities, shaping them into EPAs, building consensus on the framework, and piloting. By offering actionable insights, it empowers readers to navigate this challenging process effectively. Whether you are at the outset of a new EPA initiative or seeking refinement of previously developed EPAs, this chapter provides a practical roadmap for establishing purposeful EPAs that contribute to robust and relevant curricula.

This chapter introduces how entrustable professional activities (EPAs) can be implemented in the pre-licensure stage of education (e.g., undergraduate health professions education), with a focus on how trainees can contribute to patient care in a legitimate participatory role prior to licensure. The relative freedom of trainees from workplace-based productivity provides pre-licensure training with a great deal of flexibility. Such flexibility allows creativity and different approaches to laying the canonical foundation for specific EPAs, preparing trainees for clinical training based on EPAs and entrustment, and establishing an appreciation for how entrustment represents an assurance of readiness for safe patient care rather than a grade. Educators can leverage the required knowledge, skills, and attitudes detailed in EPA descriptions to ensure that foundational science courses are preparing trainees for clinical care. The transition from canonical knowledge to more EPA-focused activities is ideally via a ‘Z-shaped’ curriculum. Factors of a trainee’s trustworthiness, which underlie entrustment in later clinical stages of training, can be explicitly introduced, explored, and strengthened in pre-workplace-based settings. Lastly, the chapter delineates how early pre-licensure workplace-based learning founded on EPAs provides opportunities to introduce contextual competencies, while advanced workplace-based education further refines practice with limited supervision, preparing trainees for the transition to additional post-licensure training or licensed practice.

The purpose of this study is to report on the design, implementation, use and evaluation of TELLme, an online platform with annotated multiple-choice questions (MCQs) for formative assessment of knowledge acquisition covering an entire undergraduate medical curriculum. We used the Educational Design Research (EDR) framework to develop TELLme as an online platform with 24/7 access for students in a co-creation process between educators, faculty and students. EDR cycle 1 focused on prototyping, while EDR cycle 2 focused on upscaling, usage analysis and evaluation. Database entries were analysed for TELLme usage. Online surveys evaluated platform usability and support for student learning. At the end of EDR cycle 2, TELLme contained 6,713 fully annotated MCQs aligned with the cognitive learning objectives of the curriculum. Up to 600 students per day used TELLme to self-assess their knowledge, answering 3,168,622 MCQs in two semesters. Surveys indicated good usability of TELLme, 75% of students indicated that TELLme helped them to identify knowledge gaps, and 72% agreed that TELLme supported their learning. Medical students use TELLme extensively to formatively assess their learning in a self-directed manner across the entire curriculum. TELLme aligns with the existing summative and formative system of assessment of the study programme.