LGLucas Garay

Lucas Garay

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Lucas Garay has a total of 21 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Analysis of longitudinal social media for monitoring symptoms during a pandemic

    by Shixu Lin, Lucas Garay, Yining Hua, Zhijiang Guo, Wanxin Li, Minghui Li, Yujie Zhang, Xiaolin Xu, Jie Yang


  • Open Access

    VaxEval: A Benchmark Dataset for Evaluating Large Language Models on Vaccine Knowledge (Preprint)

    by Siyuan Chen, Zhengdong Wu, Lily Wass, Lucas Garay, José Vizoso, Kathy Leung, Leesa Lin

    Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Large Language Models (LLMs) are increasingly utilized in medical knowledge platforms, yet their capabilities in specialized domains like vaccination remain underexplored. Understanding their performance in these domains is vital for their successful application in educational and public health initiatives. Objective: The objective of this study is to evaluate the performance of leading Large Language Models (LLMs) in the specialized domain of vaccination knowledge using a newly developed Vaccination Evaluation Benchmark Dataset. By assessing their accuracy across multiple-choice questions (MCQs) pertaining to 14 common mandatory or recommended vaccines, this study aims to identify the strengths and weaknesses of LLMs in vaccine-related knowledge. Additionally, it seeks to provide insights into the selection and optimization of LLMs for use in public health initiatives and medical education, with a focus on improving their reliability and effectiveness in underrepresented vaccine areas. Methods: This study generates the Vaccination Evaluation Benchmark Dataset, comprising 1,341 multiple-choice questions (MCQs) to assess the vaccine knowledge of various leading LLMs, including GPT-4o, Claude3-Opus, and Gemini 1.5 Pro. The dataset draws from global health authoritative sources such as the WHO and CDC for 918 MCQs and peer-reviewed scientific literature from January 2015 to April 2024 for an additional 423 MCQs, covering 14 common mandatory or recommended vaccines such as COVID-19, Dengue, Polio, and RSV. Results: The GPT-4o model demonstrated the highest accuracy at 88.6% (95% CI: 87.0-90.3). Other models like Claude 3, GPT-4, and Gemini Pro showed competitive results in the mid-80s, while older models such as GPT-3.5 and Zero-one AI achieved accuracies below 80%. LLMs exhibited robust knowledge in well-publicized vaccine areas like Influenza and COVID-19, yet performed weaker in lesser-known vaccine categories such as RSV and Dengue. Conclusions: This study underscores the critical importance of a robust selection process for LLMs in developing digital health tools. Newer models, with more comprehensive training, consistently outperform older ones, particularly in vaccination-related knowledge. To maximize their impact on public health and medical education, selecting the right models and enhancing their training on less-publicized vaccines is essential for creating accurate, reliable, and effective tools in addressing global health challenges.

  • Clinical Text Datasets for Medical Artificial Intelligence and Large Language Models — A Systematic Review

    by Jiageng Wu, Xiaocong Liu, Minghui Li, Wanxin Li, Zichang Su, Shixu Lin, Lucas Garay, Zhiyun Zhang, Yujie Zhang, Qingcheng Zeng, et al.