Lucas Garay
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Lucas Garay has a total of 21 co-authors. Here are the top co-authors by number of shared publications:
- Jie Yang
Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women’s Hospital, Boston, MA, USA
2 shared publications - Minghui Li
School of Public Health, Zhejiang University School of Medicine, Hangzhou 310058 China
2 shared publications - Shixu Lin
School of Public Health, Zhejiang University School of Medicine, Hangzhou 310058 China
2 shared publications - Wanxin Li
School of Public Health, The Second Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China
2 shared publications - Yujie Zhang
School of Public Health, Zhejiang University School of Medicine, Hangzhou 310058 China
2 shared publications - Changzheng Yuan
School of Public Health, Second Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China
1 shared publication
Latest Publications
- 18 January 2025
- Published by Elsevier in Journal of Biomedical Informatics
- Vol. 162, 104778
Analysis of longitudinal social media for monitoring symptoms during a pandemic
- 18 January 2025
- Published by Elsevier in Journal of Biomedical Informatics
- Vol. 162, 104778
Analysis of longitudinal social media for monitoring symptoms during a pandemic
- Open Access
- 13 January 2025
- Published by JMIR Publications Inc. in JMIR AI
VaxEval: A Benchmark Dataset for Evaluating Large Language Models on Vaccine Knowledge (Preprint)
Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Large Language Models (LLMs) are increasingly utilized in medical knowledge platforms, yet their capabilities in specialized domains like vaccination remain underexplored. Understanding their performance in these domains is vital for their successful application in educational and public health initiatives. Objective: The objective of this study is to evaluate the performance of leading Large Language Models (LLMs) in the specialized domain of vaccination knowledge using a newly developed Vaccination Evaluation Benchmark Dataset. By assessing their accuracy across multiple-choice questions (MCQs) pertaining to 14 common mandatory or recommended vaccines, this study aims to identify the strengths and weaknesses of LLMs in vaccine-related knowledge. Additionally, it seeks to provide insights into the selection and optimization of LLMs for use in public health initiatives and medical education, with a focus on improving their reliability and effectiveness in underrepresented vaccine areas. Methods: This study generates the Vaccination Evaluation Benchmark Dataset, comprising 1,341 multiple-choice questions (MCQs) to assess the vaccine knowledge of various leading LLMs, including GPT-4o, Claude3-Opus, and Gemini 1.5 Pro. The dataset draws from global health authoritative sources such as the WHO and CDC for 918 MCQs and peer-reviewed scientific literature from January 2015 to April 2024 for an additional 423 MCQs, covering 14 common mandatory or recommended vaccines such as COVID-19, Dengue, Polio, and RSV. Results: The GPT-4o model demonstrated the highest accuracy at 88.6% (95% CI: 87.0-90.3). Other models like Claude 3, GPT-4, and Gemini Pro showed competitive results in the mid-80s, while older models such as GPT-3.5 and Zero-one AI achieved accuracies below 80%. LLMs exhibited robust knowledge in well-publicized vaccine areas like Influenza and COVID-19, yet performed weaker in lesser-known vaccine categories such as RSV and Dengue. Conclusions: This study underscores the critical importance of a robust selection process for LLMs in developing digital health tools. Newer models, with more comprehensive training, consistently outperform older ones, particularly in vaccination-related knowledge. To maximize their impact on public health and medical education, selecting the right models and enhancing their training on less-publicized vaccines is essential for creating accurate, reliable, and effective tools in addressing global health challenges.
- 13 January 2025
- Published by JMIR Publications Inc. in JMIR AI
Open AccessVaxEval: A Benchmark Dataset for Evaluating Large Language Models on Vaccine Knowledge (Preprint)
Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Large Language Models (LLMs) are increasingly utilized in medical knowledge platforms, yet their capabilities in specialized domains like vaccination remain underexplored. Understanding their performance in these domains is vital for their successful application in educational and public health initiatives. Objective: The objective of this study is to evaluate the performance of leading Large Language Models (LLMs) in the specialized domain of vaccination knowledge using a newly developed Vaccination Evaluation Benchmark Dataset. By assessing their accuracy across multiple-choice questions (MCQs) pertaining to 14 common mandatory or recommended vaccines, this study aims to identify the strengths and weaknesses of LLMs in vaccine-related knowledge. Additionally, it seeks to provide insights into the selection and optimization of LLMs for use in public health initiatives and medical education, with a focus on improving their reliability and effectiveness in underrepresented vaccine areas. Methods: This study generates the Vaccination Evaluation Benchmark Dataset, comprising 1,341 multiple-choice questions (MCQs) to assess the vaccine knowledge of various leading LLMs, including GPT-4o, Claude3-Opus, and Gemini 1.5 Pro. The dataset draws from global health authoritative sources such as the WHO and CDC for 918 MCQs and peer-reviewed scientific literature from January 2015 to April 2024 for an additional 423 MCQs, covering 14 common mandatory or recommended vaccines such as COVID-19, Dengue, Polio, and RSV. Results: The GPT-4o model demonstrated the highest accuracy at 88.6% (95% CI: 87.0-90.3). Other models like Claude 3, GPT-4, and Gemini Pro showed competitive results in the mid-80s, while older models such as GPT-3.5 and Zero-one AI achieved accuracies below 80%. LLMs exhibited robust knowledge in well-publicized vaccine areas like Influenza and COVID-19, yet performed weaker in lesser-known vaccine categories such as RSV and Dengue. Conclusions: This study underscores the critical importance of a robust selection process for LLMs in developing digital health tools. Newer models, with more comprehensive training, consistently outperform older ones, particularly in vaccination-related knowledge. To maximize their impact on public health and medical education, selecting the right models and enhancing their training on less-publicized vaccines is essential for creating accurate, reliable, and effective tools in addressing global health challenges.
- 23 May 2024
- Published by Massachusetts Medical Society in NEJM AI
- Vol. 1 (6)
Clinical Text Datasets for Medical Artificial Intelligence and Large Language Models — A Systematic Review
- 23 May 2024
- Published by Massachusetts Medical Society in NEJM AI
- Vol. 1 (6)
Clinical Text Datasets for Medical Artificial Intelligence and Large Language Models — A Systematic Review