MHMichael Holcomb

Michael Holcomb

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Michael Holcomb has a total of 21 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Prospective deployment of multimodal AI grading for medical student OSCEs

    by Huong-Tra Ngo, Ameer Hamza Shakur, Michael Holcomb, Shinyoung Kang, David Hein, Judah Gruen, Hunter Schuler, Philip Jarrett, Thomas Dalton, Krystle Campbell, et al.

  • Open Access

    Automated Assessment of OSCE Physical Exams using Multimodal AI

    by Shinyoung Kang, Michael J. Holcomb, Ameer H. Shakur, David Hein, Huong-Tra Ngo, Hunter Schuler, Philip C. Jarrett, Thomas O. Dalton, Andrew R. Jamieson

    Background: The assessment of physical examination skills in medical education is resource-intensive and prone to inter-rater variability. While artificial intelligence (AI) has successfully automated the grading of clinical notes and transcripts, evaluating the physical techniques themselves—what students do rather than what they say—remains an unsolved challenge. We evaluated whether a multimodal AI system could assess physical examination skills with expert-level reliability. Methods: In this retrospective ablation study, we analyzed 300 video-recorded encounters from six Objective Structured Clinical Examination (OSCE) stations (cardiovascular, respiratory, gastrointestinal, musculoskeletal, and neurological). We compared the performance of a multimodal AI model (Gemini 2.5 Pro) across single- and multi-camera configurations and isolated input modalities (video, audio-only, transcript-only, visual-only) against standard human grading. The primary outcome was agreement with a physician-adjudicated ground-truth reference standard, measured by quadratic weighted Cohen’s kappa (κ). Results: The AI system using a synchronized 3-camera native video configuration achieved significantly higher reliability (κ = 0.830; 95% CI, 0.773–0.880) than the standard human evaluators (κ = 0.732; 95% CI, 0.687–0.776). Performance followed a strict hierarchy: native video > audio-only > transcript-only > visual-only. Notably, visual-only models failed (κ ≈ 0.20) despite high detection accuracy, revealing a “visual paradox” where models could identify when an action occurred but not how well it was performed without audio cues. Conclusions: A properly configured multimodal AI system can grade physical examination skills with reliability exceeding that of trained human evaluators. Success requires native processing of synchronized audio-visual streams; transcript-based or visual-only approaches are insufficient for high-stakes assessment. These findings suggest that AI can provide scalable, objective, and valid assessment of clinical skills, overcoming the limitations of traditional human grading. Abstract Figure:


  • Open Access

    Physical Examination Identification in Medical Education Videos: Zero-Shot Multimodal AI With Temporal Sequence Optimization Study

    by Shinyoung Kang, Michael Holcomb, David Hein, Ameer Hamza Shakur, Thomas Dalton, Andrew Jamieson

    Background: Objective structured clinical examinations (OSCEs) are widely used for assessing medical student competency, but their evaluation is resource-intensive, requiring trained evaluators to review 15-minute videos. The physical examination (PE) component typically constitutes only a small portion of these recordings; yet, current automated approaches struggle with processing long medical videos due to computational constraints and difficulties maintaining temporal context. Objective: This study aims to determine whether multimodal large language models (MM-LLMs) can effectively segment PE periods within OSCE videos without previous training, potentially reducing the evaluation burden on both human graders and automated assessment systems. Methods: We analyzed 500 videos from 5 OSCE stations at University of Texas Southwestern Simulation Center, each 15 minutes long, by using hand-labeled PE periods as ground truth. Frames were sampled at 1, 2, or 3 seconds. A pose detection preprocessing step filtered frames without people. Six MM-LLMs performed frame-level classification into encounter states by using a standardized prompt. To enforce temporal consistency, we used a hidden Markov model with Viterbi decoding, merging states into 3 primary activities (consulting/notes, physical examination, and no doctor) and adding a brief edge buffer to avoid truncating true PE segments. Performance was computed per video and averaged across the dataset by using recall, precision, intersection over union (IOU), and predicted PE length with 95% CIs. Results: At 1-second sampling, GPT-4o achieved recall of 0.998 (95% CI 0.994-1.000), IOU of 0.784 (95% CI 0.765-0.803), and precision of 0.792 (95% CI 0.774-0.811), identifying a mean of 175 (SD 83) seconds of content per video as PE versus a mean labeled PE of 126 (SD 61) seconds, yielding an 81% reduction in video needing review (from 900 to 175 seconds). Across stations, recall remained high, with expected IOU variability linked to examination format and camera geometry. Increasing the sampling interval modestly decreased recall while slightly improving IOU and precision. Comparative baselines (eg, Gemini 2.0 Flash, Gemma 3, and Qwen2.5-VL variants) demonstrated trade-offs between recall and overselection; GPT-4o offered the best balance among high-recall models. Error analysis highlighted false negatives during occluded or verbally guided maneuvers and false positives during preparatory actions, suggesting opportunities for camera placement optimization and multimodal fusion (eg, audio cues). Conclusions: Integrating zero-shot MM-LLMs with minimal-supervision temporal modeling effectively segments PE periods in OSCE videos without requiring extensive training data. This approach significantly reduces review time while maintaining clinical assessment integrity, demonstrating that artificial intelligence methods combining zero-shot capabilities and light supervision can be optimized for medical education’s specific requirements. This technique establishes a foundation for more efficient and scalable clinical skill assessment across diverse medical education settings.


  • Open Access

    Physical Examination Identification in Medical Education Videos: Zero-Shot Multimodal AI With Temporal Sequence Optimization Study (Preprint)

    by Shinyoung Kang, Michael Holcomb, David Hein, Ameer Hamza Shakur, Thomas Dalton, Andrew Jamieson

    BACKGROUND Objective structured clinical examinations (OSCEs) are widely used for assessing medical student competency, but their evaluation is resource-intensive, requiring trained evaluators to review 15-minute videos. The physical examination (PE) component typically constitutes only a small portion of these recordings; yet, current automated approaches struggle with processing long medical videos due to computational constraints and difficulties maintaining temporal context. OBJECTIVE This study aims to determine whether multimodal large language models (MM-LLMs) can effectively segment PE periods within OSCE videos without previous training, potentially reducing the evaluation burden on both human graders and automated assessment systems. METHODS We analyzed 500 videos from 5 OSCE stations at University of Texas Southwestern Simulation Center, each 15 minutes long, by using hand-labeled PE periods as ground truth. Frames were sampled at 1, 2, or 3 seconds. A pose detection preprocessing step filtered frames without people. Six MM-LLMs performed frame-level classification into encounter states by using a standardized prompt. To enforce temporal consistency, we used a hidden Markov model with Viterbi decoding, merging states into 3 primary activities (consulting/notes, physical examination, and no doctor) and adding a brief edge buffer to avoid truncating true PE segments. Performance was computed per video and averaged across the dataset by using recall, precision, intersection over union (IOU), and predicted PE length with 95% CIs. RESULTS At 1-second sampling, GPT-4o achieved recall of 0.998 (95% CI 0.994-1.000), IOU of 0.784 (95% CI 0.765-0.803), and precision of 0.792 (95% CI 0.774-0.811), identifying a mean of 175 (SD 83) seconds of content per video as PE versus a mean labeled PE of 126 (SD 61) seconds, yielding an 81% reduction in video needing review (from 900 to 175 seconds). Across stations, recall remained high, with expected IOU variability linked to examination format and camera geometry. Increasing the sampling interval modestly decreased recall while slightly improving IOU and precision. Comparative baselines (eg, Gemini 2.0 Flash, Gemma 3, and Qwen2.5-VL variants) demonstrated trade-offs between recall and overselection; GPT-4o offered the best balance among high-recall models. Error analysis highlighted false negatives during occluded or verbally guided maneuvers and false positives during preparatory actions, suggesting opportunities for camera placement optimization and multimodal fusion (eg, audio cues). CONCLUSIONS Integrating zero-shot MM-LLMs with minimal-supervision temporal modeling effectively segments PE periods in OSCE videos without requiring extensive training data. This approach significantly reduces review time while maintaining clinical assessment integrity, demonstrating that artificial intelligence methods combining zero-shot capabilities and light supervision can be optimized for medical education’s specific requirements. This technique establishes a foundation for more efficient and scalable clinical skill assessment across diverse medical education settings. CLINICALTRIAL

  • Open Access

    Automatic Physical Examination Segmentation within Objective Structured Clinical Examination Videos

    by Shinyoung Kang, Michael J. Holcomb, David Hein, Ameer Hamza Shakur, Thomas O. Dalton, Andrew R. Jamieson

    Objective: Assessing medical student performance in Objective Structured Clinical Examinations (OSCEs) is labor-intensive, requiring trained evaluators to review 15-minute long videos. The ‘physical examination’ period constitutes only a small portion of these videos. Automated segmentation of OSCE videos could significantly streamline the evaluation process by detecting this physical exam portion for targeted evaluation. Current video analysis approaches struggle with these long recordings due to computational constraints and challenges in maintaining temporal context. This study tests whether multimodal large language models (MM-LLMs) can segment physical examination periods in OSCE videos without prior training, potentially easing the burden on both human graders and automated systems. Methods: We analyzed 500 videos from five OSCE stations at UT Southwestern Simulation Center, each 15 minutes long, using hand-labeled physical examination periods as ground truth. MM-LLMs processed video frames at one frame per second, classifying them into discrete activity states. A hidden Markov model with Viterbi decoding ensured temporal consistency across segments, addressing the inherent challenges of frame-by-frame classification. Results: Using Viterbi decoding trained on just 50 hand-labeled videos (10 from each station), zero-shot GPT-4o achieved 99.8% recall and 78.3% intersection over union (IOU), effectively capturing physical examinations with an average duration of 175 seconds from 900-second videos—an 81% reduction in frames requiring review. Conclusions: Integrating multimodal large language models with temporal modeling effectively segments physical examination periods in OSCE videos without requiring extensive training data. This approach significantly reduces review time while maintaining clinical assessment integrity, demonstrating that zero-shot AI methods can be optimized for medical education’s specific requirements. The technique establishes a foundation for more efficient and scalable clinical skills assessment across diverse medical education settings.


  • Rubrics to Prompts: Assessing Medical Student Post-Encounter Notes with AI

    by Andrew R. Jamieson, Michael J. Holcomb, Thomas O. Dalton, Krystle K. Campbell, Sol Vedovato, Ameer Hamza Shakur, Shinyoung Kang, David Hein, Jack Lawson, Gaudenz Danuser, et al.


  • Open Access

    A Silver Lining? Fewer Non–Severe Acute Respiratory Syndrome Coronavirus 2 Respiratory Viruses During the Coronavirus Disease 2019 Pandemic

    by Zachary M Most, Michael Holcomb, Andrew R Jamieson, Trish M Perl

    Non-pharmaceutical interventions (NPIs) have “flattened the curve” of the COVID-19 pandemic, however the effect of these interventions on other respiratory viruses is unknown. We used aggregate level case count data for eight respiratory viruses and compared the institutional and statewide case counts before and during the period that NPIs were active. We observed a 61% (IRR 0.39, 95% CI 0.37 to 0.41, P < 0.0001) decrease in non-SARS-CoV-2 respiratory viral infections when NPIs were implemented. This finding, if further verified, should guide future public health initiatives to mitigate viral epidemics.