Aly A. Valliani

BACKGROUND AND OBJECTIVES: Multiple-choice questions are the primary assessment format for neurosurgical board certification. Creating high-quality examination questions requires significant expert time and resources. The goal of this study was to develop an automated system to generate board-style neurosurgical multiple-choice questions using state-of-the-art vision-language models and compare their quality with authentic self-assessment questions. METHODS: We developed an automated pipeline using OpenAI generative pre-trained transformer (GPT)-4o and Anthropic Claude Sonnet-3.5 to generate neurosurgical board-style questions from Neurosurgery Publications articles. We generated 89 587 synthetic questions: 45 689 with GPT-4o and 43 898 with Claude. Each question was associated with a single image extracted from the articles' figures. We evaluated the quality of synthetic questions through 5 surveys comparing 20 synthetic questions (10 from each model) with 10 authentic questions from the Self-Assessment for Neurological Surgeons (SANS) question bank. Each survey was completed by a neurosurgery resident and an attending who guessed the source [human vs artificial intelligence (AI)-generated] and rated suitability for board examination use. We also evaluated the question-answering performance of the generalist GPT-4o and the specialized CNS-Obsidian. RESULTS: SANS questions were more often perceived as human-made than GPT-generated (residents, P = .0002; attendings, P = .1091) and Claude-generated (residents, P = .0002; attendings, P = .0272) questions. Notably, 54% of AI-generated questions misled at least one evaluator, and 23% misled both. In quality assessments, SANS questions outperformed GPT-generated (residents, P < 10 ⁻⁵ ; attendings, P = .0001) and Claude-generated (residents and attendings, P < 10 ⁻⁵ ) questions. Particularly, 25% of AI-generated questions were rated as suitable for board examinations vs 72% of human-generated questions when measured by evaluator consensus ( P < 10 ⁻⁷ ). CONCLUSION: Although quality gaps exist between AI-generated and human-created neurosurgical board examination questions, our approach demonstrates the potential of vision-language models to augment assessment development in specialized medical fields, reducing the burden on examination boards and credentialing organizations.

BACKGROUND AND OBJECTIVES: Myxopapillary ependymomas (MPE) and intradural lumbosacral schwannomas may be challenging to distinguish based on presenting characteristics and preoperative imaging. Accurate differentiation is crucial, as MPEs carry a risk of cerebrospinal fluid dissemination and warrant earlier intervention, a more tailored surgical strategy, consideration for adjuvant radiation, and frequent surveillance. Here, we describe our institutional experience with these tumors and develop a radiomics-based machine learning model to help distinguish them on preoperative imaging. METHODS: Institutional surgical records from 2011 to 2025 were queried and clinical data were extracted for the retrospective cohort analysis. Tumors were manually segmented in ITK-Snap from T1 postcontrast images, and radiomics features were extracted using the PyRadiomics package. An ensemble of random forest, k-nearest neighbors, and naive Bayes classifiers was trained on a subset of radiomics features using nested cross-validation. RESULTS: Our cohort included 101 cases, including 32 MPEs, 61 intradural schwannomas, and 8 dumbbell schwannomas with a circumscribed intradural component. Twenty-four consecutive tumors (3 MPEs and 21 schwannomas) were used as a held-out pseudoprospective test set. No significant difference in presenting International Standards for Neurological Classification of Spinal Cord Injury grade was observed ( P = .558). Our radiomics model incorporated segmentations with inter-rater DICE scores >0.83 for all tumors. Cross-validation (area under the receiver operating characteristic = 0.895) and test (area under the receiver operating characteristic = 0.984) discriminatory performance was high and identified biologically relevant features, such as maximum 3-dimensional diameter, that were significantly increased ( P < .001) in MPEs, likely due to longitudinal tumor growth along the filum. Excluding scoliotic patients did not significantly alter discrimination, suggesting robustness to vertebral column malalignment that may coexist with intradural tumors. CONCLUSION: A radiomics-based machine learning model demonstrated excellent discriminative ability between MPE and lumbosacral schwannoma, achieving high accuracy and robustness to vertebral alignment variations. These results suggest that radiomics-based models may be developed into a useful tool for preoperative planning and patient counseling.

Background: Given global population growth and aging, it is imperative to prioritize early eye disease detection and treatment. However, as patient volume increases, providers are facing a shortage of workforce capacity, particularly in areas where eye doctors are already scarce, making it important to consider alternative innovative solutions that could help increase eye screening capabilities. This study compared virtual reality (VR) platform of vision screening exams that are used to evaluate ocular health, such as 24-2 perimetry, Ishihara tiles, and the Amsler grid, against their in-clinic counterparts. Methods: A total of 86 subjects were recruited from Mount Sinai’s ophthalmology clinic (New York, USA) for a comparison trial that was internally controlled across healthy eyes and those with glaucoma and retinal diseases. VR and in-office tests were administered to the patients during their clinical visit, including 24-2 perimetry, Ishihara tiles, and the Amsler grid in a randomized order, and the results were compared for each test. Results: Perimetry results from Humphrey Visual Field Analyzer (HVFA) and VR suprathreshold testing demonstrated a good sensitivity both overall (80% OD, 84% OS) and across control (86% OD, 89% OS), glaucoma (69% OD, 78% OS), and retinal disease (76% OD, 80% OS) groups. A Garway-Heath anatomical map showed an overall 70–80% agreement. Ishihara plate tests did not show a significant difference between the two testing modalities (p = 0.12; Mann–Whitney U test), which remained true across all groups. Amsler grid testing differences were also non-significant within each subgroup (p = 0.81; Mann–Whitney U test). Patient time required to complete VR exams was significantly improved (p < 0.0001; Welch’s t-test) compared to the clinical standard tests. Conclusions: All VR-based exams tested in this study showed high sensitivity and percent agreement when compared to their in-office standards. Given the results of this study, VR has a promising potential in visual function screening, which, in addition to its portable design and easy use, could assist eye doctors in screening for prevalent diseases such as glaucoma and retinal conditions. Translational Relevance: VR-based vision exams that test vision fields, color vision and visual distortions provide comparable results in healthy patients, as well as those with glaucoma and retinal diseases, indicating its potential as a screening technology for different ocular pathologies. Given VR’s portable and low-profile features, it is important to consider leveraging VR to augment delivery of vision care.

BACKGROUND AND OBJECTIVES: Traditional medical board examinations present clinical information in static vignettes with multiple-choices (MC), fundamentally different from how physicians gather and integrate data in practice. Recent advances in large language models (LLMs) offer promising approaches to creating more realistic clinical interactive conversations. However, these approaches are limited in neurosurgery, where patient communication capacity varies significantly and diagnosis heavily relies on objective data such as imaging and neurological examinations. We aimed to develop and evaluate a multi–artificial intelligence (AI) agent conversation framework for neurosurgical case assessment that enables realistic clinical interactions through simulated patients and structured access to objective clinical data. METHODS: We developed a framework to convert 608 Self-Assessment in Neurological Surgery first-order diagnosis questions into conversation sessions using 3 specialized AI agents: patient AI for subjective information, system AI for objective data, and clinical AI for diagnostic reasoning. We evaluated generative pretrained transformer 4o's (GPT-4o's) diagnostic accuracy across traditional vignettes, patient-only conversations, and patient + system AI interactions, with human benchmark testing from 10 neurosurgery residents. RESULTS: GPT-4o showed significant performance drops from traditional vignettes to conversational formats in both MC (89.0%-60.9%, P < .0001) and free-response scenarios (78.4%-30.3%, P < .0001). Adding access to objective data through system AI improved performance (to 67.4%, P = .0015; and 61.8%, P < .0001, respectively). Questions requiring image interpretation showed similar patterns but lower accuracy. Residents outperformed GPT-4o in free-response conversations (70.0% vs 28.3%, P = .0030) using fewer interactions and reported high educational value of the interactive format. CONCLUSION: This multi-AI agent framework provides both a more challenging evaluation method for LLMs and an engaging educational tool for neurosurgical training. The significant performance drops in conversational formats suggest that traditional MC testing may overestimate LLMs' clinical reasoning capabilities, while the framework's interactive nature offers promising applications for enhancing medical education.