Zhiyong Lu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Recognitions
Zhiyong Lu has a total of 998 co-authors. Here are the top co-authors by number of shared publications:
Chih-Hsuan Wei
Division of Intramural Research (DIR), National Library of Medicine (NLM), National Institutes of Health (NIH), Bethesda, MD 20892, USA
38 shared publications- Qingyu Chen
Department of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University, New Haven, USA
36 shared publications - Qiao Jin
National Library of Medicine, National Institutes of Health, Bethesda, USA
31 shared publications Robert Leaman
National Library of Medicine (NLM), National Institutes of Health (NIH), MD, 20894, Bethesda, USA
28 shared publications- Yifan Peng
Department of Population Health Sciences, Weill Cornell Medicine, New York, USA
27 shared publications Qingyu Chen
23 shared publications
Latest Publications
- Open Access
Detecting Contradictory Evidence on Drug-Disease Associations at Scale
Contradictory claims about drug efficacy across randomized controlled trials (RCTs) complicate evidence-based medicine, yet existing automated methods classify isolated sentence pairs and do not explain why studies disagree. We present DrugCon, a large language model (LLM)-based framework that extracts study conclusions from RCT abstracts, links these claims when they concern the same drug-disease association, detects contradictions with counterfactual reasoning, and explains them using a taxonomy of ten study-level factors such as population, dosage, and outcome measures. Applying DrugCon to 224,756 RCT abstracts, we built DrugCon-Bench, an exhaustively annotated benchmark of 542 contradictory claim pairs from 819 abstracts. Most contradictions involve more than one study-level factor (75.5%). The most common are differences in outcome measures (64.0%), populations (51.3%), and dosage or exposure duration (35.6%). DrugCon achieves an F1 score of 0.744 for contradiction detection, outperforming rule-based, supervised, and natural language inference baselines, and an F1 of 0.482 for identifying explanatory study factors. DrugCon enables scalable and explainable analysis of contradictory clinical evidence.
Open AccessDetecting Contradictory Evidence on Drug-Disease Associations at Scale
Contradictory claims about drug efficacy across randomized controlled trials (RCTs) complicate evidence-based medicine, yet existing automated methods classify isolated sentence pairs and do not explain why studies disagree. We present DrugCon, a large language model (LLM)-based framework that extracts study conclusions from RCT abstracts, links these claims when they concern the same drug-disease association, detects contradictions with counterfactual reasoning, and explains them using a taxonomy of ten study-level factors such as population, dosage, and outcome measures. Applying DrugCon to 224,756 RCT abstracts, we built DrugCon-Bench, an exhaustively annotated benchmark of 542 contradictory claim pairs from 819 abstracts. Most contradictions involve more than one study-level factor (75.5%). The most common are differences in outcome measures (64.0%), populations (51.3%), and dosage or exposure duration (35.6%). DrugCon achieves an F1 score of 0.744 for contradiction detection, outperforming rule-based, supervised, and natural language inference baselines, and an F1 of 0.482 for identifying explanatory study factors. DrugCon enables scalable and explainable analysis of contradictory clinical evidence.
- Open Access
- 28 October 2026
- Published by Springer Nature in Nature Communications
Supervising the search process produces reliable and generalizable information-seeking agents
Large language models (LLMs) are transforming web search by shifting from document ranking to synthesizing answers, and are increasingly deployed as autonomous agentic search systems that iteratively interact with external knowledge sources. Despite this progress, building effective search agents remains challenging because high-quality intermediate search steps are difficult to generate. Previous approaches have primarily relied on outcome supervision, rewarding agents only for producing correct final answers. This often leads to reward hacking and excessive dependence on parametric memory, limiting generalization to out-of-domain tasks. To address these limitations, we introduce RAG-Gym, a framework that shifts supervision from final answers to the search process itself. With RAG-Gym, we systematically investigate architecture design, parameter optimization, and action evaluation, identifying reasoning reflection as a critical capability for search agents. Building on this insight, we propose Re 2 Search++, a process-supervised agent that achieves substantial improvements on multi-hop information-seeking benchmarks, especially in out-of-domain settings. Performance gains are driven primarily by higher-quality search queries rather than answer optimization alone, and the learned search critics transfer across models, including proprietary LLMs. These findings show that supervising the search process produces more reliable and generalizable information-seeking agents.
- 28 October 2026
- Published by Springer Nature in Nature Communications
Open AccessSupervising the search process produces reliable and generalizable information-seeking agents
Large language models (LLMs) are transforming web search by shifting from document ranking to synthesizing answers, and are increasingly deployed as autonomous agentic search systems that iteratively interact with external knowledge sources. Despite this progress, building effective search agents remains challenging because high-quality intermediate search steps are difficult to generate. Previous approaches have primarily relied on outcome supervision, rewarding agents only for producing correct final answers. This often leads to reward hacking and excessive dependence on parametric memory, limiting generalization to out-of-domain tasks. To address these limitations, we introduce RAG-Gym, a framework that shifts supervision from final answers to the search process itself. With RAG-Gym, we systematically investigate architecture design, parameter optimization, and action evaluation, identifying reasoning reflection as a critical capability for search agents. Building on this insight, we propose Re 2 Search++, a process-supervised agent that achieves substantial improvements on multi-hop information-seeking benchmarks, especially in out-of-domain settings. Performance gains are driven primarily by higher-quality search queries rather than answer optimization alone, and the learned search critics transfer across models, including proprietary LLMs. These findings show that supervising the search process produces more reliable and generalizable information-seeking agents.
- Open Access
- 19 September 2026
- Published by Springer Nature in Communications Medicine
Assessing safety and trustworthiness of large language models in medicine
Background: The remarkable capabilities of large language models make them increasingly compelling for use in real-world healthcare applications. However, the risks associated with using these artificial intelligence systems in medicine are not systematically understood. The aim of this study is to characterize these risks by applying five key principles for safe and trustworthy medical artificial intelligence: truthfulness, resilience, fairness, robustness, and privacy. Methods: We introduce MedGuard-Bench: a safety benchmark featuring one thousand expert-verified questions covering ten specific aspects of our five core principles. We use this comprehensive corpus to systematically evaluate sixteen commonly used large language models, assessing their safety and reliability in medical contexts. Results: We show that current large language models generally perform poorly on most of our safety tests, regardless of their safety alignment mechanisms. Our evaluation demonstrates that these models fall significantly short when compared to the high performance and reliability of human physicians. Conclusions: Despite reports indicating that advanced large language models can match or exceed human performance in various medical tasks, this study reveals a significant safety gap in current technology. This underscores the crucial need for ongoing human oversight and the implementation of strict safety guardrails before deploying these tools in clinical practice.
- 19 September 2026
- Published by Springer Nature in Communications Medicine
Open AccessAssessing safety and trustworthiness of large language models in medicine
Background: The remarkable capabilities of large language models make them increasingly compelling for use in real-world healthcare applications. However, the risks associated with using these artificial intelligence systems in medicine are not systematically understood. The aim of this study is to characterize these risks by applying five key principles for safe and trustworthy medical artificial intelligence: truthfulness, resilience, fairness, robustness, and privacy. Methods: We introduce MedGuard-Bench: a safety benchmark featuring one thousand expert-verified questions covering ten specific aspects of our five core principles. We use this comprehensive corpus to systematically evaluate sixteen commonly used large language models, assessing their safety and reliability in medical contexts. Results: We show that current large language models generally perform poorly on most of our safety tests, regardless of their safety alignment mechanisms. Our evaluation demonstrates that these models fall significantly short when compared to the high performance and reliability of human physicians. Conclusions: Despite reports indicating that advanced large language models can match or exceed human performance in various medical tasks, this study reveals a significant safety gap in current technology. This underscores the crucial need for ongoing human oversight and the implementation of strict safety guardrails before deploying these tools in clinical practice.
- Open Access
- 26 August 2026
- Published by Elsevier in Ophthalmology Science
- Vol. 6 (11)
Opportunistic Cardiovascular Risk Assessment in the Eye Clinic: Validation of a Retinal Deep Learning–Based 10-Year Atherosclerotic Cardiovascular Disease Estimator
- 26 August 2026
- Published by Elsevier in Ophthalmology Science
- Vol. 6 (11)
Open AccessOpportunistic Cardiovascular Risk Assessment in the Eye Clinic: Validation of a Retinal Deep Learning–Based 10-Year Atherosclerotic Cardiovascular Disease Estimator
- Open Access
- 8 August 2026
- Published by Association for Computing Machinery (ACM)
- p. 11774-11785
Condition-Gated Reasoning for Context-Dependent Biomedical Question Answering
Current biomedical question answering (QA) systems often assume that medical knowledge applies uniformly, yet real-world clinical reasoning is inherently conditional: nearly every decision depends on patient-specific factors such as comorbidities and contraindications. Existing benchmarks do not evaluate such conditional reasoning, and retrieval-augmented or graph-based methods lack explicit mechanisms to ensure that retrieved knowledge is applicable to given context. To address this gap, we propose CondMedQA, the first benchmark for conditional biomedical QA, consisting of multi-hop questions whose answers vary with patient conditions. Furthermore, we propose Condition-Gated Reasoning (CGR), a novel framework that constructs condition-aware knowledge graphs and selectively activates or prunes reasoning paths based on query conditions. Our findings show that CGR more reliably selects condition-appropriate answers while matching or exceeding state-of-the-art performance on biomedical QA benchmarks, highlighting the importance of explicitly modeling conditionality for robust medical reasoning.
- 8 August 2026
- Published by Association for Computing Machinery (ACM)
- p. 11774-11785
Open AccessCondition-Gated Reasoning for Context-Dependent Biomedical Question Answering
Current biomedical question answering (QA) systems often assume that medical knowledge applies uniformly, yet real-world clinical reasoning is inherently conditional: nearly every decision depends on patient-specific factors such as comorbidities and contraindications. Existing benchmarks do not evaluate such conditional reasoning, and retrieval-augmented or graph-based methods lack explicit mechanisms to ensure that retrieved knowledge is applicable to given context. To address this gap, we propose CondMedQA, the first benchmark for conditional biomedical QA, consisting of multi-hop questions whose answers vary with patient conditions. Furthermore, we propose Condition-Gated Reasoning (CGR), a novel framework that constructs condition-aware knowledge graphs and selectively activates or prunes reasoning paths based on query conditions. Our findings show that CGR more reliably selects condition-appropriate answers while matching or exceeding state-of-the-art performance on biomedical QA benchmarks, highlighting the importance of explicitly modeling conditionality for robust medical reasoning.
- 24 July 2026
- Published by Springer Nature in Nature Protocols
- p. 1-12
Tutorial: guidance on the use of large language models for medical research
- 24 July 2026
- Published by Springer Nature in Nature Protocols
- p. 1-12
Tutorial: guidance on the use of large language models for medical research
- Top Cited Papers
- 2 July 2026
- Published by Springer Nature in Nature Machine Intelligence
- Vol. 8 (7) , 1142-1156
Empowering biomedical evidence exploration and synthesis with deep knowledge graph research
- 2 July 2026
- Published by Springer Nature in Nature Machine Intelligence
- Vol. 8 (7) , 1142-1156
Top Cited PapersEmpowering biomedical evidence exploration and synthesis with deep knowledge graph research
- Open Access
- 1 July 2026
- Published by Oxford University Press (OUP) in Bioinformatics
- Vol. 42 (Supplement)
Knowledge-guided contextual gene set analysis with large language models.
Gene set analysis (GSA) is a foundational approach for interpreting genomic data of diseases by linking genes to biological processes. However, conventional GSA methods overlook clinical context of the analyses, often generating long lists of enriched pathways with redundant, nonspecific, or irrelevant results. Interpreting these requires extensive, ad-hoc manual effort, reducing both reliability and reproducibility. We introduce cGSA, a novel AI-driven framework that enhances GSA by incorporating context-aware pathway prioritization. cGSA integrates gene cluster detection, enrichment analysis, and large language models to identify pathways that are not only statistically significant but also biologically meaningful. Benchmarking on 102 curated gene sets across 19 diseases and ten disease-related biological mechanisms shows that cGSA outperforms baseline methods by over 30%, with expert validation confirming its increased precision and interpretability. Two independent case studies in melanoma and breast cancer further demonstrate its potential to uncover context-specific insights and support targeted hypothesis. The demo website is publicly available at https://www.ncbi.nlm.nih.gov/CBBresearch/Lu/Demo/cGSA/, while the data and code can be accessed at https://github.com/ncbi-nlp/cGSA.
- 1 July 2026
- Published by Oxford University Press (OUP) in Bioinformatics
- Vol. 42 (Supplement)
Open AccessKnowledge-guided contextual gene set analysis with large language models.
Gene set analysis (GSA) is a foundational approach for interpreting genomic data of diseases by linking genes to biological processes. However, conventional GSA methods overlook clinical context of the analyses, often generating long lists of enriched pathways with redundant, nonspecific, or irrelevant results. Interpreting these requires extensive, ad-hoc manual effort, reducing both reliability and reproducibility. We introduce cGSA, a novel AI-driven framework that enhances GSA by incorporating context-aware pathway prioritization. cGSA integrates gene cluster detection, enrichment analysis, and large language models to identify pathways that are not only statistically significant but also biologically meaningful. Benchmarking on 102 curated gene sets across 19 diseases and ten disease-related biological mechanisms shows that cGSA outperforms baseline methods by over 30%, with expert validation confirming its increased precision and interpretability. Two independent case studies in melanoma and breast cancer further demonstrate its potential to uncover context-specific insights and support targeted hypothesis. The demo website is publicly available at https://www.ncbi.nlm.nih.gov/CBBresearch/Lu/Demo/cGSA/, while the data and code can be accessed at https://github.com/ncbi-nlp/cGSA.
- Top Cited PapersOpen Access
- 22 June 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Computing for Healthcare
- Vol. 7 (3) , 1-38
LMOD \(\boldsymbol{+}\) : A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in OphthalmologyThe rising prevalence of vision-threatening eye diseases poses a major global health and economic burden, yet timely diagnosis remains limited by workforce shortages, diagnostic delays, and restricted access to specialized care. Artificial intelligence (AI) offers potential solutions. In particular, recent progress in foundation models and large language models—especially multimodal large language models (MLLMs)—has shown promise in medical image interpretation and automated clinical documentation. However, advancing MLLMs for ophthalmology is hindered by the lack of unified, comprehensive benchmark datasets for development and evaluation. Most existing benchmarks were designed for earlier models, which focused on narrow tasks or specific disease conditions. These benchmarks typically provide outputs in the form of disease labels rather than free-text responses. As a result, they are less suitable for assessing emerging generative models. In this work, we present LMOD+, a large-scale multimodal ophthalmology benchmark dataset comprising 32,633 instances with multi-granular annotations across 12 common ophthalmic conditions and 5 imaging modalities. The dataset integrates imaging, anatomical structures, demographics, and free-text annotations. It supports primary ophthalmic applications such as anatomical structure recognition, disease screening, disease staging, and demographic prediction for potential performance bias evaluation. Alongside the dataset, we introduce a systematic and unified data curation pipeline that repurposes existing or new datasets for MLLM development. LMOD+ extends our preliminary LMOD benchmark—the first multimodal ophthalmology benchmark for MLLMs—with three major enhancements. First, we expanded the dataset by nearly 50% (from 21,933 to 32,633 instances). The color fundus photography (CFP) modality, the most accessible imaging modality in ophthalmology, was significantly enlarged to cover a broader range of pathological conditions. Second, we broadened task coverage to include (a) 12 binary disease diagnosis tasks for prevalent conditions such as diabetic retinopathy, age-related macular degeneration, and retinal vein occlusion; (b) multi-class ophthalmic disease diagnosis; (c) disease severity classification, including a diabetic retinopathy staging task, which uses two internationally adopted grading standards: the international clinical diabetic retinopathy classification and the Scottish diabetic retinopathy grading scheme classification; and (d) demographic prediction (age and sex) to assess potential model bias. Third, we systematically evaluated 24 state-of-the-art MLLMs, including recent models from the InternVL, Qwen, and DeepSeek families. Our evaluations highlight both the promise and limitations of current MLLMs in ophthalmology. For example, Qwen-7B and InternVL achieved accuracies of 58.26% and 57.83% in disease screening under a zero-shot setting with a single model—a considerably more challenging paradigm than traditional fine-tuning, where separate models are trained for each specific task. InternVL also demonstrated potential in anatomical recognition. Nonetheless, overall performance remained suboptimal and often close to random baselines for challenging tasks such as disease staging, underscoring the substantial gap between general-domain MLLMs and the specialized requirements of ophthalmology. We publicly release the dataset, curation pipeline, and leaderboard to encourage community-wide development and evaluation of MLLMs, with the goal of advancing ophthalmic applications and ultimately reducing the global burden of vision-threatening diseases through AI. The dataset website, benchmark leaderboard, and download link are available at https://kfzyqin.github.io/lmod_plus .
- 22 June 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Computing for Healthcare
- Vol. 7 (3) , 1-38
Top Cited PapersOpen AccessLMOD \(\boldsymbol{+}\) : A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in OphthalmologyThe rising prevalence of vision-threatening eye diseases poses a major global health and economic burden, yet timely diagnosis remains limited by workforce shortages, diagnostic delays, and restricted access to specialized care. Artificial intelligence (AI) offers potential solutions. In particular, recent progress in foundation models and large language models—especially multimodal large language models (MLLMs)—has shown promise in medical image interpretation and automated clinical documentation. However, advancing MLLMs for ophthalmology is hindered by the lack of unified, comprehensive benchmark datasets for development and evaluation. Most existing benchmarks were designed for earlier models, which focused on narrow tasks or specific disease conditions. These benchmarks typically provide outputs in the form of disease labels rather than free-text responses. As a result, they are less suitable for assessing emerging generative models. In this work, we present LMOD+, a large-scale multimodal ophthalmology benchmark dataset comprising 32,633 instances with multi-granular annotations across 12 common ophthalmic conditions and 5 imaging modalities. The dataset integrates imaging, anatomical structures, demographics, and free-text annotations. It supports primary ophthalmic applications such as anatomical structure recognition, disease screening, disease staging, and demographic prediction for potential performance bias evaluation. Alongside the dataset, we introduce a systematic and unified data curation pipeline that repurposes existing or new datasets for MLLM development. LMOD+ extends our preliminary LMOD benchmark—the first multimodal ophthalmology benchmark for MLLMs—with three major enhancements. First, we expanded the dataset by nearly 50% (from 21,933 to 32,633 instances). The color fundus photography (CFP) modality, the most accessible imaging modality in ophthalmology, was significantly enlarged to cover a broader range of pathological conditions. Second, we broadened task coverage to include (a) 12 binary disease diagnosis tasks for prevalent conditions such as diabetic retinopathy, age-related macular degeneration, and retinal vein occlusion; (b) multi-class ophthalmic disease diagnosis; (c) disease severity classification, including a diabetic retinopathy staging task, which uses two internationally adopted grading standards: the international clinical diabetic retinopathy classification and the Scottish diabetic retinopathy grading scheme classification; and (d) demographic prediction (age and sex) to assess potential model bias. Third, we systematically evaluated 24 state-of-the-art MLLMs, including recent models from the InternVL, Qwen, and DeepSeek families. Our evaluations highlight both the promise and limitations of current MLLMs in ophthalmology. For example, Qwen-7B and InternVL achieved accuracies of 58.26% and 57.83% in disease screening under a zero-shot setting with a single model—a considerably more challenging paradigm than traditional fine-tuning, where separate models are trained for each specific task. InternVL also demonstrated potential in anatomical recognition. Nonetheless, overall performance remained suboptimal and often close to random baselines for challenging tasks such as disease staging, underscoring the substantial gap between general-domain MLLMs and the specialized requirements of ophthalmology. We publicly release the dataset, curation pipeline, and leaderboard to encourage community-wide development and evaluation of MLLMs, with the goal of advancing ophthalmic applications and ultimately reducing the global burden of vision-threatening diseases through AI. The dataset website, benchmark leaderboard, and download link are available at https://kfzyqin.github.io/lmod_plus .
- Open Access
- 10 June 2026
- Published by Springer Nature in Scientific Data
- Vol. 13 (1)
A Curated Dataset for Question Answering on Laboratory Test Reference Ranges and Interpretation
Laboratory tests are crucial for diagnosing and managing health conditions, offering essential reference ranges for result interpretation. The diversity of lab tests, influenced by variables such as the specimen type (e.g., blood, urine), gender, age, and other influencing factors like pregnancy, makes automated interpretation challenging. Automated clinical decision support systems attempting to interpret these values must account for such nuances to avoid misdiagnoses or incorrect clinical decisions. In this regard, we present LabQAR (Laboratory Question Answering with Reference Ranges), a manually curated dataset comprising 550 lab test reference ranges derived from authoritative medical sources, encompassing 363 unique lab tests and including multiple-choice questions with annotations on reference ranges, specimen types, and other factors impacting interpretation. We also assess the performance of several large language models (LLMs), including LLaMA 3.1, GatorTronGPT, GPT-3.5 turbo, GPT-4 turbo, and GPT-4o, in predicting reference ranges and classifying results as normal, low, or high. The findings indicate that GPT-4o outperforms other models, showcasing the potential of LLMs in clinical decision support.
- 10 June 2026
- Published by Springer Nature in Scientific Data
- Vol. 13 (1)
Open AccessA Curated Dataset for Question Answering on Laboratory Test Reference Ranges and Interpretation
Laboratory tests are crucial for diagnosing and managing health conditions, offering essential reference ranges for result interpretation. The diversity of lab tests, influenced by variables such as the specimen type (e.g., blood, urine), gender, age, and other influencing factors like pregnancy, makes automated interpretation challenging. Automated clinical decision support systems attempting to interpret these values must account for such nuances to avoid misdiagnoses or incorrect clinical decisions. In this regard, we present LabQAR (Laboratory Question Answering with Reference Ranges), a manually curated dataset comprising 550 lab test reference ranges derived from authoritative medical sources, encompassing 363 unique lab tests and including multiple-choice questions with annotations on reference ranges, specimen types, and other factors impacting interpretation. We also assess the performance of several large language models (LLMs), including LLaMA 3.1, GatorTronGPT, GPT-3.5 turbo, GPT-4 turbo, and GPT-4o, in predicting reference ranges and classifying results as normal, low, or high. The findings indicate that GPT-4o outperforms other models, showcasing the potential of LLMs in clinical decision support.