Maxwell B. Singer

The rapid growth of biomedical literature poses challenges for manual knowledge curation and synthesis. Biomedical Natural Language Processing (BioNLP) automates the process. While Large Language Models (LLMs) have shown promise in general domains, their effectiveness in BioNLP tasks remains unclear due to limited benchmarks and practical guidelines. We perform a systematic evaluation of four LLMs—GPT and LLaMA representatives—on 12 BioNLP benchmarks across six applications. We compare their zero-shot, few-shot, and fine-tuning performance with the traditional fine-tuning of BERT or BART models. We examine inconsistencies, missing information, hallucinations, and perform cost analysis. Here, we show that traditional fine-tuning outperforms zero- or few-shot LLMs in most tasks. However, closed-source LLMs like GPT-4 excel in reasoning-related tasks such as medical question answering. Open-source LLMs still require fine-tuning to close performance gaps. We find issues like missing information and hallucinations in LLM outputs. These results offer practical insights for applying LLMs in BioNLP.

Background: We previously demonstrated that retinal ischemic perivascular lesions (RIPLs), which are indicative of ischemia in the middle retina, may be a biomarker of ischemic cardiovascular disease. In this study, we sought to determine the relationship between RIPLs and atrial fibrillation, a common source of cardiac emboli. Methods and Results: In this case‐control study, we identified individuals between the ages of 50 and 90 years who had undergone macular spectral domain optical coherence tomography imaging. Individuals with atrial fibrillation were identified, and age‐ and sex‐matched individuals from the same pool, but without a diagnosis of atrial fibrillation, were selected as controls. Spectral domain optical coherence tomography scans were reviewed by 3 independent and masked observers for presence of RIPLs. The relationship between RIPLs and atrial fibrillation was analyzed using multivariable logistic regression models. There were 106 and 91 subjects with and without atrial fibrillation, respectively. The percentage of subjects with RIPLs was higher in the atrial fibrillation group compared with the control group (57.5% versus 37.4%; P =0.005). After adjusting for age, sex, smoking history, hypertension, diabetes, coronary artery disease, carotid stenosis, stroke, and myocardial infarction, the presence of RIPLs was significantly associated with atrial fibrillation, with an odds ratio of 1.91 (95% CI, 1.01–3.59). Conclusions: RIPLs are significantly associated with atrial fibrillation, independent of underlying ischemic heart disease or cardiovascular risk factors. This association may inform the diagnostic cardiovascular workup for individuals with RIPLs incidentally detected on optical coherence tomography scan of the macula.