Adam B. Wilson

Practice pointsOn deductive coding tasks, LLMs’ independent coding of excerpts matched human agreement with consensus.Sequential independent LLM coding showed human-comparable agreement, whereas ba...

Objectives: The increasing volume of medical education research necessitates efficient, reliable, and scalable methods for conducting quality appraisals. The Medical Education Research Study Quality Instrument (MERSQI) is a widely used tool, although its manual scoring process remains resource‐intensive. This study evaluated how well large language models (LLMs) appraise medical education research using the MERSQI tool in comparison with human judges. Methods: Three LLMs (GPT‐5, Claude Sonnet 4, and Gemini 2.5 Pro) assigned MERSQI domain scores to 1423 medical education research articles. The authors compared AI‐generated scores with human‐generated scores using intraclass correlation coefficients (ICCs) across the six MERSQI domains. They evaluated the agreement between AI‐ and human‐generated MERSQI composite scores using Bland–Altman plots. Results: Domain‐level ICC values ranged from fair (0.24) to near perfect (0.81), with the lowest agreement observed in the ‘sampling,’ ‘validity evidence,’ and ‘data analysis’ domains. No single LLM consistently outperformed the others across all domains. Composite score agreement with human ratings was substantial and similar across LLMs (ICC range: 0.65–0.69). GPT‐5 produced slightly lower composite scores than humans, while Claude Sonnet 4 and Gemini 2.5 Pro produced higher scores, with Gemini showing the largest deviation. The Bland–Altman plots for Gemini 2.5 Pro suggested proportional bias, indicating its agreement with human scores varied across the range of study quality. Conclusions: These LLMs demonstrated substantial agreement with human raters for MERSQI composite scores, but domain‐level agreement varied. Systematic differences in scoring patterns highlight the need for human oversight and additional calibrations before integrating LLMs into systematic review appraisal workflows.

Peer Instruction (PI) has demonstrated benefits in STEM education, but its effectiveness as a distinct instructional method in health professions education remains uncertain. This systematic review identified 17 eligible studies implementing Mazur's PI framework in medical and dental education, with 12 contributing to meta-analyses. PI yielded large within-session conceptual gains (Hedges' g ≥ 1.48; p < 0.001) but showed no performance superiority over alternative instructional approaches (p = 0.082). 74% of learners reported PI to be helpful (p < 0.001). Evidence for long-term retention or transfer remains limited. Curricular adoption should be guided by logistical feasibility and learning objective alignment rather than expectations of superior performance over other established active strategies. The online version contains supplementary material available at 10.1007/s40670-026-02744-1.

Purpose This study meta-analyzed associations between academic prematriculation metrics (undergraduate GPA and MCAT scores) and performance outcomes (preclerkship and board examination performance) among US allopathic and osteopathic medical students. Given the evolving landscape of medical school admissions, we examined whether these associations have changed over time. Methods Following PRISMA guidelines, we systematically searched MEDLINE, Scopus, Google Scholar, and ERIC for studies published before June 2023. Inclusion criteria required studies to report correlations between prematriculation metrics and medical school preclerkship performance (e.g., first-year GPA, preclerkship GPA) or board examination scores (e.g., USMLE Step 1, COMLEX-USA Level 1). Random-effects meta-analyses estimated pooled correlation coefficients, with subgroup analyses assessing differences by medical program type and publication era. Results A total of 23 studies met the inclusion criteria, encompassing a maximum of 4965 students across the 14 analyses. All prematriculation metrics showed statistically significant but weak positive correlations with preclerkship performance and board examination scores (R_pooled = 0.22–0.37). MCAT Biological Science and Total scores had the strongest associations with USMLE Step 1 scores, while science GPA had weaker associations. No significant trend in correlation strength was observed over time. Conclusions These findings support using holistic admissions review practices to account for non-academic attributes. Future research should explore alternative associations or predictors and assess their impact on longer-term clinical performance.

Medical schools would benefit from systematic guidance for developing comprehensive artificial intelligence (AI) policies, given generative AI's rapid integration into medical education. This study developed and applied an idealized AI policy framework to analyze AI-related documents at U.S. medical school institutions, providing reference points for the development and refinement of institutional policies. AI-related documents from institutions with U.S. allopathic and osteopathic medical schools were systematically collected (from August to October 2024) and analyzed using a comprehensive framework containing 24 subthemes across six themes: Background/Context, Governance, AI Literacy, Tools/Usage, Ethical/Legal Considerations, and Technology Support and Infrastructure. Publicly available online documents were systematically coded to generate framework subtheme scores indicating breadth of coverage across framework themes. AI-related documents retrieved from 73.7% (146/198) of U.S. medical school institutions covered an average of 8 of 24 subthemes, representing a mean framework coverage score of 32.3% ± 19.8 Rarely addressed subthemes included Audit and Compliance Mechanisms (6.8%, 10/146), Technical Infrastructure (6.2%, 9/146), and Environmental Stewardship (1.4%, 2/146). Academic Honesty and Plagiarism dominated AI-related documents (81.5%, 119/146), followed by Decision-Making Authority (54.1%, 79/146) and Critical Evaluation (52.1%, 76/146). Formal AI policies demonstrated significantly higher framework coverage than other AI document types (44.0% vs 30.4%, p = 0.003). Seven institutions with the highest coverage (≥13/24 subthemes) shared seven common distinguishing features, with six present universally. AI-related documents currently emphasize academic integrity over strategic planning, with substantial gaps in infrastructure and review mechanisms. Institutions can enhance their AI policies by incorporating common features identified in well-designed policies and following frameworks that strike a balance between immediate concerns and long-term adaptability.

Purpose Universal Design for Learning (UDL) is an educational approach that aims to promote access to information and a method for supporting students, whether they identify as having a disability or not. There is little evidence of UDL approaches being incorporated into health professions programming. Materials and Methods This survey-based study evaluated health professions faculty and administrators from 30 institutions regarding their perceptions and understanding of UDL principles and availability of UDL resources. Results At least 88.2% (45/51) of respondents indicated it was important/very important to give students access to information using different modalities, adjustable formats, and assistive technologies. Over half (54.9%; 28/51) indicated faculty are primarily responsible for ensuring access to UDL resources. However, 80.4% (41/51) indicated increased faculty workload is a barrier to maximizing UDL resources. Many (60.8%; 31/51) agreed/strongly agreed that maximizing UDL resources is prioritized by their institutions, but only 39.2% (20/51) agreed/strongly agreed their institution provided sufficient financial/human resources. Conclusions Our results suggest there is disparity between the interest for and institutional support of UDL resources. This study underscores the need for health professions programming related to education and implementation of UDL principles within existing curricula to proactively support all students in an inclusive manner.

Evidence suggests custom chatbots are superior to commercial generative artificial intelligence (GenAI) systems for text‐based anatomy content inquiries. This study evaluates ChatGPT‐4o's and Claude 3.5 Sonnet's capabilities to interpret unlabeled anatomical images. Secondarily, ChatGPT o1‐preview was evaluated as an AI rater to grade AI‐generated outputs using a rubric and was compared against human raters. Anatomical images (five musculoskeletal, five thoracic) representing diverse image‐based media (e.g., illustrations, photographs, MRI) were annotated with identification markers (e.g., arrows, circles) and uploaded to each GenAI system for interpretation. Forty‐five prompts (i.e., 15 first‐order, 15 second‐order, and 15 third‐order questions) with associated images were submitted to both GenAI systems across two timepoints. Responses were graded by anatomy experts for factual accuracy and superfluity (the presence of excessive wording) on a three‐point Likert scale. ChatGPT o1‐preview was tested for agreement against human anatomy experts to determine its usefulness as an AI rater. Statistical analyses included inter‐rater agreement, hierarchical linear modeling, and test–retest reliability. ChatGPT‐4o's factual accuracy score across 45 outputs was 68.0% compared to Claude 3.5 Sonnet's score of 61.5% (p = 0.319). As an AI rater, ChatGPT o1‐preview showed moderate to substantial agreement with human raters (Cohen's kappa = 0.545–0.755) for evaluating factual accuracy according to a rubric of textbook answers. Further improvements and evaluations are needed before commercial GenAI systems can be used as credible student resources in anatomy education. Similarly, ChatGPT o1‐preview demonstrates promise as an AI assistant for educational research, though further investigation is warranted.