David Kurland

BACKGROUND AND OBJECTIVES: Multiple-choice questions are the primary assessment format for neurosurgical board certification. Creating high-quality examination questions requires significant expert time and resources. The goal of this study was to develop an automated system to generate board-style neurosurgical multiple-choice questions using state-of-the-art vision-language models and compare their quality with authentic self-assessment questions. METHODS: We developed an automated pipeline using OpenAI generative pre-trained transformer (GPT)-4o and Anthropic Claude Sonnet-3.5 to generate neurosurgical board-style questions from Neurosurgery Publications articles. We generated 89 587 synthetic questions: 45 689 with GPT-4o and 43 898 with Claude. Each question was associated with a single image extracted from the articles' figures. We evaluated the quality of synthetic questions through 5 surveys comparing 20 synthetic questions (10 from each model) with 10 authentic questions from the Self-Assessment for Neurological Surgeons (SANS) question bank. Each survey was completed by a neurosurgery resident and an attending who guessed the source [human vs artificial intelligence (AI)-generated] and rated suitability for board examination use. We also evaluated the question-answering performance of the generalist GPT-4o and the specialized CNS-Obsidian. RESULTS: SANS questions were more often perceived as human-made than GPT-generated (residents, P = .0002; attendings, P = .1091) and Claude-generated (residents, P = .0002; attendings, P = .0272) questions. Notably, 54% of AI-generated questions misled at least one evaluator, and 23% misled both. In quality assessments, SANS questions outperformed GPT-generated (residents, P < 10 ⁻⁵ ; attendings, P = .0001) and Claude-generated (residents and attendings, P < 10 ⁻⁵ ) questions. Particularly, 25% of AI-generated questions were rated as suitable for board examinations vs 72% of human-generated questions when measured by evaluator consensus ( P < 10 ⁻⁷ ). CONCLUSION: Although quality gaps exist between AI-generated and human-created neurosurgical board examination questions, our approach demonstrates the potential of vision-language models to augment assessment development in specialized medical fields, reducing the burden on examination boards and credentialing organizations.

BACKGROUND AND OBJECTIVES: The objective of this study was to compare sociodemographic factors, presenting characteristics, and outcomes between cohorts of patients receiving surgery for spinal metastases at 2 neighboring institutions, 1 private and 1 public, affiliated with a single major academic medical center in a large metropolitan area. METHODS: This analysis included all patients who underwent decompressive surgery for extradural spinal metastases. Sociodemographic factors, treatment characteristics, and outcomes were compared between those treated at a private hospital and a neighboring public hospital using Rao-Scott χ ² tests for categorical variables, Student t tests for continuous variables, and the Kaplan-Meier product-limit method for overall survival and progression-free survival. RESULTS: Compared with those treated at our private hospital, patients treated at our public hospital were more often younger ( P = .005), of Black or Hispanic race (72.6% vs 19%, P < .001), and uninsured (16% vs 5.6%, P = .005). They more frequently presented with epidural spinal cord compression grade 3 (76% vs 56.8%, P = .027), were nonambulatory before surgery (56.9% vs 13.5%, P < .001), and had increased neurological impairment as denoted by American Spinal Injury Association Impairment Scale grades of A, B, or C (39.2% vs 7.5%). Patients treated at our public hospital had shorter median follow-up time (92 vs 302.5 days, P = .004). Multivariate analysis did not reveal a significant difference in overall survival or progression-free survival between hospitals, instead demonstrating associations with primary tumor histology and number of spinal metastases ( P < .05). CONCLUSION: There were substantial disparities in sociodemographic factors, presenting local disease burden, and postoperative neurological outcome but no difference in survival outcome, between patients treated at our public and private hospitals. These findings underscore the need for more equitable screening, surveillance, and referral structures.

BACKGROUND AND OBJECTIVES: Traditional medical board examinations present clinical information in static vignettes with multiple-choices (MC), fundamentally different from how physicians gather and integrate data in practice. Recent advances in large language models (LLMs) offer promising approaches to creating more realistic clinical interactive conversations. However, these approaches are limited in neurosurgery, where patient communication capacity varies significantly and diagnosis heavily relies on objective data such as imaging and neurological examinations. We aimed to develop and evaluate a multi–artificial intelligence (AI) agent conversation framework for neurosurgical case assessment that enables realistic clinical interactions through simulated patients and structured access to objective clinical data. METHODS: We developed a framework to convert 608 Self-Assessment in Neurological Surgery first-order diagnosis questions into conversation sessions using 3 specialized AI agents: patient AI for subjective information, system AI for objective data, and clinical AI for diagnostic reasoning. We evaluated generative pretrained transformer 4o's (GPT-4o's) diagnostic accuracy across traditional vignettes, patient-only conversations, and patient + system AI interactions, with human benchmark testing from 10 neurosurgery residents. RESULTS: GPT-4o showed significant performance drops from traditional vignettes to conversational formats in both MC (89.0%-60.9%, P < .0001) and free-response scenarios (78.4%-30.3%, P < .0001). Adding access to objective data through system AI improved performance (to 67.4%, P = .0015; and 61.8%, P < .0001, respectively). Questions requiring image interpretation showed similar patterns but lower accuracy. Residents outperformed GPT-4o in free-response conversations (70.0% vs 28.3%, P = .0030) using fewer interactions and reported high educational value of the interactive format. CONCLUSION: This multi-AI agent framework provides both a more challenging evaluation method for LLMs and an engaging educational tool for neurosurgical training. The significant performance drops in conversational formats suggest that traditional MC testing may overestimate LLMs' clinical reasoning capabilities, while the framework's interactive nature offers promising applications for enhancing medical education.