Karl L. Sangwon

Operational decisions governing patient flow, cost, and quality of care demand specialized predictive models, yet most clinical NLP efforts focus on medical knowledge benchmarks. We introduce Lang1, a family of language models (100M-7B parameters) pretrained on 80 billion clinical tokens from NYU Langone Health electronic health records blended with 627 billion internet tokens. We evaluate Lang1 on the REalistic Medical Evaluation (ReMedE), an evaluation suite derived from 668,331 Electronic Health Records (EHR) notes spanning five tasks: readmission, mortality prediction, length of stay, comorbidity coding, and insurance denial. In zero-shot settings, both general-purpose and biomedical models underperform on four of five tasks. After finetuning, Lang1-1B outperforms finetuned generalist models up to 70x larger and zero-shot models up to 671x larger. Joint multi-task finetuning yields cross-task transfer, and Lang1-1B transfers effectively to unseen tasks and an external health system. These results demonstrate that effective healthcare AI requires in-domain pretraining, supervised finetuning, and evaluation beyond proxy benchmarks.

BACKGROUND AND OBJECTIVES: Traditional medical board examinations present clinical information in static vignettes with multiple-choices (MC), fundamentally different from how physicians gather and integrate data in practice. Recent advances in large language models (LLMs) offer promising approaches to creating more realistic clinical interactive conversations. However, these approaches are limited in neurosurgery, where patient communication capacity varies significantly and diagnosis heavily relies on objective data such as imaging and neurological examinations. We aimed to develop and evaluate a multi–artificial intelligence (AI) agent conversation framework for neurosurgical case assessment that enables realistic clinical interactions through simulated patients and structured access to objective clinical data. METHODS: We developed a framework to convert 608 Self-Assessment in Neurological Surgery first-order diagnosis questions into conversation sessions using 3 specialized AI agents: patient AI for subjective information, system AI for objective data, and clinical AI for diagnostic reasoning. We evaluated generative pretrained transformer 4o's (GPT-4o's) diagnostic accuracy across traditional vignettes, patient-only conversations, and patient + system AI interactions, with human benchmark testing from 10 neurosurgery residents. RESULTS: GPT-4o showed significant performance drops from traditional vignettes to conversational formats in both MC (89.0%-60.9%, P < .0001) and free-response scenarios (78.4%-30.3%, P < .0001). Adding access to objective data through system AI improved performance (to 67.4%, P = .0015; and 61.8%, P < .0001, respectively). Questions requiring image interpretation showed similar patterns but lower accuracy. Residents outperformed GPT-4o in free-response conversations (70.0% vs 28.3%, P = .0030) using fewer interactions and reported high educational value of the interactive format. CONCLUSION: This multi-AI agent framework provides both a more challenging evaluation method for LLMs and an engaging educational tool for neurosurgical training. The significant performance drops in conversational formats suggest that traditional MC testing may overestimate LLMs' clinical reasoning capabilities, while the framework's interactive nature offers promising applications for enhancing medical education.

BACKGROUND AND OBJECTIVES: Hand-sewn (HS) microsuturing is limited by tremor and fatigue. Robotic systems may improve performance, but quantitative comparisons remain limited. This study evaluates the precision of robot-assisted vs manual suturing during end-to-side microanastomosis. METHODS: Under simulation, microsurgical robot-assisted and HS sutures for end-to-side microanastomosis were performed by a single surgeon. One thousand four hundred and ninety-four total frames were assessed within 4 videos. Each robotic clip was paired with a corresponding HS clip. Tracker software extracted 2-dimensional positional data. Motion trajectories were smoothed using Savitzky-Golay filtering for an ideal suture trajectory. Deviation from an ideal path was quantified using Euclidean distance. Mean and SD of path deviation were calculated. Group comparisons were made as fold-change reductions and standardized effect sizes (Cohen d) to quantify the magnitude of observed differences. RESULTS: Robotic-assisted sutures demonstrated significantly lower mean path deviation and variability than HS sutures, particularly for the dominant (right) hand, with consistently large effect sizes for the right hand (all = 1.2, r = 0.5) and smaller for the left hand (d = 0.36-0.71, r = 0.18-0.33). CONCLUSION: Robotic microsuturing with microsurgical assistant significantly improves path fidelity, particularly in dominant-hand tasks. Manual sutures showed larger deviations between the ideal suture and raw data, supporting robotic integration into cerebrovascular neurosurgery and warranting study in live models.

Hospitals and healthcare systems rely on operational decisions that determine patient flow, cost, and quality of care. Despite strong performance on medical knowledge and conversational benchmarks, foundation models trained on general text may lack the specialized knowledge required for these operational decisions. We introduce Lang1, a family of models (100M-7B parameters) pretrained on a specialized corpus blending 80B clinical tokens from NYU Langone Health's EHRs and 627B tokens from the internet. To rigorously evaluate Lang1 in real-world settings, we developed the REalistic Medical Evaluation (ReMedE), a benchmark derived from 668,331 EHR notes that evaluates five critical tasks: 30-day readmission prediction, 30-day mortality prediction, length of stay, comorbidity coding, and predicting insurance claims denial. In zero-shot settings, both general-purpose and specialized models underperform on four of five tasks (36.6%-71.7% AUROC), with mortality prediction being an exception. After finetuning, Lang1-1B outperforms finetuned generalist models up to 70x larger and zero-shot models up to 671x larger, improving AUROC by 3.64%-6.75% and 1.66%-23.66% respectively. We also observed cross-task scaling with joint finetuning on multiple tasks leading to improvement on other tasks. Lang1-1B effectively transfers to out-of-distribution settings, including other clinical tasks and an external health system. Our findings suggest that predictive capabilities for hospital operations require explicit supervised finetuning, and that this finetuning process is made more efficient by in-domain pretraining on EHR. Our findings support the emerging view that specialized LLMs can compete with generalist models in specialized tasks, and show that effective healthcare systems AI requires the combination of in-domain pretraining, supervised finetuning, and real-world evaluation beyond proxy benchmarks.

BACKGROUND AND OBJECTIVES: Trans-sonolucent cranioplasty ultrasonography (TCUS) has been explored as a noninvasive tool for evaluating superficial temporal artery (STA)–middle cerebral artery (MCA) bypass patency. Previous research has focused on early postoperative feasibility. Data on its long-term utility and correlation with formal angiography remain scarce. We aimed to evaluate TCUS' role in postoperative STA-MCA bypass graft monitoring and its concordance with formal angiography. METHODS: This retrospective study included 46 consecutive direct STA-MCA anastomoses in 40 patients (March 2021-May 2024), all with sonolucent polymethyl methacrylate cranioplasty. Patient records were reviewed for demographics, disease and surgical characteristics, and outcomes. Postoperative TCUS was performed outpatient to monitor anastomotic patency. Formal follow-up angiography was also conducted, and radiographic data were reviewed for graft patency assessment and qualitative correlation with TCUS. RESULTS: Follow-up angiography was performed for 41 of 46 anastomoses (digital subtraction angiography, n = 34; computed tomography angiography, n = 4; magnetic resonance angiography, n = 3) at a median of 1.1 years, demonstrating 97.6% patency (40/41). Outpatient TCUS was performed in 32 of 46 bypasses (69.6%) with 100% patency at first scan (median 28.5 days). A second TCUS (n = 19, 41.3%) at a median of 8.4 months (3.9-13.6 months) showed robust flow in 94.7% of cases. One bypass had asymptomatic slow flow with a narrowed anastomosis, and another showed a severely stenosed STA correlating with later digital subtraction angiography. In the broader cohort, third (n = 5, median 1.2 years) and fourth (n = 1, 1.4 years) TCUS assessments demonstrated 100% patency. Among bypasses undergoing both TCUS and angiography (n = 31, 67.4%), findings were concordant in all cases. CONCLUSION: TCUS demonstrated complete agreement with formal angiography in assessing bypass patency, supporting TCUS as a reliable, noninvasive monitoring tool. Future research should explore quantitative TCUS flow measurements and their relationship to intraoperative flow and long-term graft remodeling.

Background & Objective: Standard clinical LLM benchmarks use multiple-choice vignettes that present all information up front, unlike real encounters where clinicians iteratively elicit histories and objective data. We hypothesized that such formats inflate LLM performance and mask weaknesses in diagnostic reasoning. We developed and evaluated a multi-AI agent conversational framework that converts JAMA Clinical Challenge cases into multi-turn dialogues, and assessed its impact on diagnostic accuracy across frontier LLMs. Methods: We adapted 815 diagnostic cases from 1,519 JAMA Clinical Challenges into two formats: (1) original vignette and (2) multi-agent conversation with a Patient AI (subjective history) and a System AI (objective data: exam, labs, imaging). A Clinical LLM queried these agents and produced a final diagnosis. Models tested were O1 (OpenAI), GPT-4o (OpenAI), LLaMA-3-70B (Meta), and Deepseek-R1-distill-LLaMA3-70B (Deepseek), each in multiple-choice and free-response modes. Free-response grading used a separate GPT-4o judge for diagnostic equivalence. Accuracy (Wilson 95% CIs) and conversation lengths were compared using two-tailed tests. Results: Accuracy decreased for all models when moving from vignettes to conversations and from multiple-choice to free-response (p<0.0001 for all pairwise comparisons). In vignette multiple-choice, accuracy was O1 79.8% (95% CI, 76.9%–82.4%), GPT-4o 74.5% (71.4%–77.4%), LLaMA-3 70.9% (69.5%–72.2%), Deepseek-R1 69.0% (67.5%–70.4%). In conversation multiple-choice: O1 69.1% (65.8%–72.2%), GPT-4o 51.3% (49.8%–52.8%), LLaMA-3 49.7% (48.2%–51.3%), Deepseek-R1 34.0% (32.6%–35.5%). In conversation free-response: O1 31.7% (28.6%–34.9%), GPT-4o 20.7% (19.5%–22.0%), LLaMA-3 22.9% (21.6%–24.2%), Deepseek-R1 9.3% (8.4%–10.2%). O1 generally required fewer conversational turns than GPT-4o, suggesting more efficient multi-turn reasoning. Conclusions: Converting vignettes into multi-agent, multi-turn dialogues reveals substantial performance drops across leading LLMs, indicating that static multiple-choice benchmarks overestimate clinical reasoning competence. Our open-source framework offers a more rigorous and discriminative evaluation and a realistic substrate for educational use, enabling assessment of iterative information-gathering and synthesis that better reflects clinical practice.

Background and Objectives: Traditional medical board examinations present clinical information in static vignettes with multiple-choices, fundamentally different from how physicians gather and integrate data in practice. Recent advances in Large Language Models (LLMs) offer promising approaches to creating more realistic clinical interactive conversations. However, these approaches are limited in neurosurgery, where patient communication capacity varies significantly and diagnosis heavily relies on objective data like imaging and neurological examinations. We aimed to develop and evaluate a multi-AI agent conversation framework for neurosurgical case assessment that enables realistic clinical interactions through simulated patients and structured access to objective clinical data. Methods: We developed a framework to convert 608 Self-Assessment in Neurological Surgery (SANS) first-order diagnosis questions into conversation sessions using three specialized AI agents: Patient AI for subjective information, System AI for objective data, and Clinical AI for diagnostic reasoning. We evaluated GPT-4o’s diagnostic accuracy across traditional vignettes, patient-only conversations, and patient+system AI interactions, with human benchmark testing from ten neurosurgery residents. Results: GPT-4o showed significant performance drops from traditional vignettes to conversational formats in both multiple-choice (89.0% to 60.9%, p<0.0001) and free-response scenarios (78.4% to 30.3%, p<0.0001). Adding access to objective data through System AI improved performance (to 67.4%, p=0.0015 and 61.8%, p<0.0001, respectively). Questions requiring image interpretation showed similar patterns but lower accuracy. Residents outperformed GPT-4o in free-response conversations (70.0% vs 28.3%, p=0.0030) using fewer interactions and reported high educational value of the interactive format. Conclusions: This multi-AI agent framework provides both a more challenging evaluation method for LLMs and an engaging educational tool for neurosurgical training. The significant performance drops in conversational formats suggest that traditional multiple-choice testing may overestimate LLMs’ clinical reasoning capabilities, while the framework’s interactive nature offers promising applications for enhancing medical education.

BACKGROUND AND OBJECTIVES: Bone metastases, affecting more than 4.8% of patients with cancer annually, and particularly spinal metastases require urgent intervention to prevent neurological complications. However, the current process of manually reviewing radiological reports leads to potential delays in specialist referrals. We hypothesized that natural language processing (NLP) review of routine radiology reports could automate the referral process for timely multidisciplinary care of spinal metastases. METHODS: We assessed 3 NLP models—a rule-based regular expression (RegEx) model, GPT-4, and a specialized Bidirectional Encoder Representations from Transformers (BERT) model (NYUTron)—for automated detection and referral of bone metastases. Study inclusion criteria targeted patients with active cancer diagnoses who underwent advanced imaging (computed tomography, MRI, or positron emission tomography) without previous specialist referral. We defined 2 separate tasks: task of identifying clinically significant bone metastatic terms (lexical detection), and identifying cases needing a specialist follow-up (clinical referral). Models were developed using 3754 hand-labeled advanced imaging studies in 2 phases: phase 1 focused on spine metastases, and phase 2 generalized to bone metastases. Standard McRae’s line performance metrics were evaluated and compared across all stages and tasks. RESULTS: In the lexical detection, a simple RegEx achieved the highest performance (sensitivity 98.4%, specificity 97.6%, F1 = 0.965), followed by NYUTron (sensitivity 96.8%, specificity 89.9%, and F1 = 0.787). For the clinical referral task, RegEx also demonstrated superior performance (sensitivity 92.3%, specificity 87.5%, and F1 = 0.936), followed by a fine-tuned NYUTron model (sensitivity 90.0%, specificity 66.7%, and F1 = 0.750). CONCLUSION: An NLP-based automated referral system can accurately identify patients with bone metastases requiring specialist evaluation. A simple RegEx model excels in syntax-based identification and expert-informed rule generation for efficient referral patient recommendation in comparison with advanced NLP models. This system could significantly reduce missed follow-ups and enhance timely intervention for patients with bone metastases.

BACKGROUND AND OBJECTIVES: Dural arteriovenous fistula (dAVF) surgery is a microsurgical procedure that requires confirmation of obliteration using formal cerebral angiography, but the lack of intraoperative angiogram or need for postoperative angiogram in some settings necessitates a search for alternative, less invasive methods to verify surgical success. This study evaluates the use of indocyanine green videoangiography FLOW 800 hemodynamic intraoperatively during cranial and spinal dAVF obliteration to confirm obliteration and predict surgical success. METHODS: A retrospective analysis was conducted using indocyanine green videoangiography FLOW 800 to intraoperatively measure 4 hemodynamic parameters—Delay Time, Speed, Time to Peak, and Rise Time—across venous drainage regions of interest pre/post-dAVF obliteration. Univariate and multivariate statistical analyses to evaluate and visualize presurgical vs postsurgical state hemodynamic changes included nonparametric statistical tests, logistic regression, and Bayesian analysis. RESULTS: A total of 14 venous drainage regions of interest from 8 patients who had successful spinal or cranial dAVF obliteration confirmed with intraoperative digital subtraction angiography were extracted. Significant hemodynamic changes were observed after dAVF obliteration, with median Speed decreasing from 13.5 to 5.5 s ⁻¹ ( P = .029) and Delay Time increasing from 2.07 to 7.86 s ( P = .020). Bayesian logistic regression identified Delay Time as the strongest predictor of postsurgical state, with a 50% increase associated with 2.16 times higher odds of achieving obliteration (odds ratio = 4.59, 95% highest density interval: 1.07-19.95). Speed exhibited a trend toward a negative association with postsurgical state (odds ratio = 0.62, 95% highest density interval: 0.26-1.42). Receiver operating characteristic–area under the curve analysis using logistic regression demonstrated a score of 0.760, highlighting Delay Time and Speed as key features distinguishing preobliteration and postobliteration states. CONCLUSION: Our findings demonstrate that intraoperative FLOW 800 analysis reliably quantifies and visualizes immediate hemodynamic changes consistent with dAVF obliteration. Speed and Delay Time emerged as key indicators of surgical success, highlighting the potential of FLOW 800 as a noninvasive adjunct to traditional imaging techniques for confirming dAVF obliteration intraoperatively.