Robert Leaman
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Robert Leaman has a total of 119 co-authors. Here are the top co-authors by number of shared publications:
- Zhiyong Lu
National Library of Medicine, National Institutes of Health
17 shared publications - Chih-Hsuan Wei
National Institutes of Health, Bethesda, Maryland, USA
8 shared publications - Qiao Jin
National Library of Medicine, National Institutes of Health, Bethesda, USA
6 shared publications - Rezarta Islamaj Doğan
National Institutes of Health, Bethesda, Maryland, USA
5 shared publications - Shubo Tian
National Library of Medicine (NLM), National Institutes of Health (NIH), Bethesda, USA
5 shared publications - Alexis Allot
Montreal Neurological Institute-Hospital, McGill University
3 shared publications
Latest Publications
- Open Access
- 19 September 2026
- Published by Springer Nature in Communications Medicine
Assessing safety and trustworthiness of large language models in medicine
Background: The remarkable capabilities of large language models make them increasingly compelling for use in real-world healthcare applications. However, the risks associated with using these artificial intelligence systems in medicine are not systematically understood. The aim of this study is to characterize these risks by applying five key principles for safe and trustworthy medical artificial intelligence: truthfulness, resilience, fairness, robustness, and privacy. Methods: We introduce MedGuard-Bench: a safety benchmark featuring one thousand expert-verified questions covering ten specific aspects of our five core principles. We use this comprehensive corpus to systematically evaluate sixteen commonly used large language models, assessing their safety and reliability in medical contexts. Results: We show that current large language models generally perform poorly on most of our safety tests, regardless of their safety alignment mechanisms. Our evaluation demonstrates that these models fall significantly short when compared to the high performance and reliability of human physicians. Conclusions: Despite reports indicating that advanced large language models can match or exceed human performance in various medical tasks, this study reveals a significant safety gap in current technology. This underscores the crucial need for ongoing human oversight and the implementation of strict safety guardrails before deploying these tools in clinical practice.
- 19 September 2026
- Published by Springer Nature in Communications Medicine
Open AccessAssessing safety and trustworthiness of large language models in medicine
Background: The remarkable capabilities of large language models make them increasingly compelling for use in real-world healthcare applications. However, the risks associated with using these artificial intelligence systems in medicine are not systematically understood. The aim of this study is to characterize these risks by applying five key principles for safe and trustworthy medical artificial intelligence: truthfulness, resilience, fairness, robustness, and privacy. Methods: We introduce MedGuard-Bench: a safety benchmark featuring one thousand expert-verified questions covering ten specific aspects of our five core principles. We use this comprehensive corpus to systematically evaluate sixteen commonly used large language models, assessing their safety and reliability in medical contexts. Results: We show that current large language models generally perform poorly on most of our safety tests, regardless of their safety alignment mechanisms. Our evaluation demonstrates that these models fall significantly short when compared to the high performance and reliability of human physicians. Conclusions: Despite reports indicating that advanced large language models can match or exceed human performance in various medical tasks, this study reveals a significant safety gap in current technology. This underscores the crucial need for ongoing human oversight and the implementation of strict safety guardrails before deploying these tools in clinical practice.
- Open Access
- 8 August 2026
- Published by Association for Computing Machinery (ACM)
- p. 11774-11785
Condition-Gated Reasoning for Context-Dependent Biomedical Question Answering
Current biomedical question answering (QA) systems often assume that medical knowledge applies uniformly, yet real-world clinical reasoning is inherently conditional: nearly every decision depends on patient-specific factors such as comorbidities and contraindications. Existing benchmarks do not evaluate such conditional reasoning, and retrieval-augmented or graph-based methods lack explicit mechanisms to ensure that retrieved knowledge is applicable to given context. To address this gap, we propose CondMedQA, the first benchmark for conditional biomedical QA, consisting of multi-hop questions whose answers vary with patient conditions. Furthermore, we propose Condition-Gated Reasoning (CGR), a novel framework that constructs condition-aware knowledge graphs and selectively activates or prunes reasoning paths based on query conditions. Our findings show that CGR more reliably selects condition-appropriate answers while matching or exceeding state-of-the-art performance on biomedical QA benchmarks, highlighting the importance of explicitly modeling conditionality for robust medical reasoning.
- 8 August 2026
- Published by Association for Computing Machinery (ACM)
- p. 11774-11785
Open AccessCondition-Gated Reasoning for Context-Dependent Biomedical Question Answering
Current biomedical question answering (QA) systems often assume that medical knowledge applies uniformly, yet real-world clinical reasoning is inherently conditional: nearly every decision depends on patient-specific factors such as comorbidities and contraindications. Existing benchmarks do not evaluate such conditional reasoning, and retrieval-augmented or graph-based methods lack explicit mechanisms to ensure that retrieved knowledge is applicable to given context. To address this gap, we propose CondMedQA, the first benchmark for conditional biomedical QA, consisting of multi-hop questions whose answers vary with patient conditions. Furthermore, we propose Condition-Gated Reasoning (CGR), a novel framework that constructs condition-aware knowledge graphs and selectively activates or prunes reasoning paths based on query conditions. Our findings show that CGR more reliably selects condition-appropriate answers while matching or exceeding state-of-the-art performance on biomedical QA benchmarks, highlighting the importance of explicitly modeling conditionality for robust medical reasoning.
- 24 July 2026
- Published by Springer Nature in Nature Protocols
- p. 1-12
Tutorial: guidance on the use of large language models for medical research
- 24 July 2026
- Published by Springer Nature in Nature Protocols
- p. 1-12
Tutorial: guidance on the use of large language models for medical research
- Open Access
- 1 July 2026
- Published by Oxford University Press (OUP) in Bioinformatics
- Vol. 42 (Supplement)
Knowledge-guided contextual gene set analysis with large language models.
Gene set analysis (GSA) is a foundational approach for interpreting genomic data of diseases by linking genes to biological processes. However, conventional GSA methods overlook clinical context of the analyses, often generating long lists of enriched pathways with redundant, nonspecific, or irrelevant results. Interpreting these requires extensive, ad-hoc manual effort, reducing both reliability and reproducibility. We introduce cGSA, a novel AI-driven framework that enhances GSA by incorporating context-aware pathway prioritization. cGSA integrates gene cluster detection, enrichment analysis, and large language models to identify pathways that are not only statistically significant but also biologically meaningful. Benchmarking on 102 curated gene sets across 19 diseases and ten disease-related biological mechanisms shows that cGSA outperforms baseline methods by over 30%, with expert validation confirming its increased precision and interpretability. Two independent case studies in melanoma and breast cancer further demonstrate its potential to uncover context-specific insights and support targeted hypothesis. The demo website is publicly available at https://www.ncbi.nlm.nih.gov/CBBresearch/Lu/Demo/cGSA/, while the data and code can be accessed at https://github.com/ncbi-nlp/cGSA.
- 1 July 2026
- Published by Oxford University Press (OUP) in Bioinformatics
- Vol. 42 (Supplement)
Open AccessKnowledge-guided contextual gene set analysis with large language models.
Gene set analysis (GSA) is a foundational approach for interpreting genomic data of diseases by linking genes to biological processes. However, conventional GSA methods overlook clinical context of the analyses, often generating long lists of enriched pathways with redundant, nonspecific, or irrelevant results. Interpreting these requires extensive, ad-hoc manual effort, reducing both reliability and reproducibility. We introduce cGSA, a novel AI-driven framework that enhances GSA by incorporating context-aware pathway prioritization. cGSA integrates gene cluster detection, enrichment analysis, and large language models to identify pathways that are not only statistically significant but also biologically meaningful. Benchmarking on 102 curated gene sets across 19 diseases and ten disease-related biological mechanisms shows that cGSA outperforms baseline methods by over 30%, with expert validation confirming its increased precision and interpretability. Two independent case studies in melanoma and breast cancer further demonstrate its potential to uncover context-specific insights and support targeted hypothesis. The demo website is publicly available at https://www.ncbi.nlm.nih.gov/CBBresearch/Lu/Demo/cGSA/, while the data and code can be accessed at https://github.com/ncbi-nlp/cGSA.
- Open Access
Cell phenotypes in the biomedical literature: a systematic analysis and text mining corpus
The variety of cell phenotypes identified by single-cell technologies is rapidly expanding, yet this knowledge is dispersed across the scientific literature and incompletely represented in structured resources. We present the CellLink corpus, a manually annotated collection of over 22,000 mentions of human and mouse cell populations in recent journal articles, distinguishing specific cell phenotypes, heterogeneous cell populations, and vague cell populations, and linking to Cell Ontology (CL) terms as either exact or related matches, covering nearly half of the terms in the current CL. A systematic analysis reveals lineage-specific patterns in how authors utilize anatomical context, molecular signatures, functional roles, developmental stage, and other attributes in cell naming. We show that fine-tuning transformer-based models on CellLink yields strong performance for named entity recognition, while embedding-based approaches support zero-shot entity linking and distinguishing exact from related matches. We further demonstrate the utility of CellLink to expand and refine the chondrocyte branch of CL.
Open AccessCell phenotypes in the biomedical literature: a systematic analysis and text mining corpus
The variety of cell phenotypes identified by single-cell technologies is rapidly expanding, yet this knowledge is dispersed across the scientific literature and incompletely represented in structured resources. We present the CellLink corpus, a manually annotated collection of over 22,000 mentions of human and mouse cell populations in recent journal articles, distinguishing specific cell phenotypes, heterogeneous cell populations, and vague cell populations, and linking to Cell Ontology (CL) terms as either exact or related matches, covering nearly half of the terms in the current CL. A systematic analysis reveals lineage-specific patterns in how authors utilize anatomical context, molecular signatures, functional roles, developmental stage, and other attributes in cell naming. We show that fine-tuning transformer-based models on CellLink yields strong performance for named entity recognition, while embedding-based approaches support zero-shot entity linking and distinguishing exact from related matches. We further demonstrate the utility of CellLink to expand and refine the chondrocyte branch of CL.
- Open Access
- 15 January 2026
- Published by Oxford University Press (OUP) in Database: The Journal of Biological Databases and Curation
- Vol. 2026
MedHopQA: a disease-centred multi-hop reasoning benchmark and evaluation framework for LLM-based biomedical question answering
Evaluating large language models (LLMs) in the biomedical domain requires benchmarks that can distinguish genuine reasoning from pattern matching and that remain discriminative as model capabilities improve. Existing biomedical question answering (QA) benchmarks are limited in this respect: multiple-choice formats allow models to succeed by answer elimination rather than inference, and widely circulated exam-style datasets are subject to performance saturation and training data contamination. Multi-hop reasoning, i.e. the ability to integrate information across multiple sources to derive an answer, is central to clinically meaningful tasks such as diagnosis support, literature-based discovery, and hypothesis generation, yet remains underrepresented in current biomedical QA benchmarks. We present MedHopQA, a disease-centred multi-hop reasoning benchmark of 1000 expert-curated question–answer pairs introduced as a shared task at BioCreative IX. Each question requires synthesis of information across two distinct Wikipedia articles, and answers are provided in open-ended free-text format rather than as multiple-choice selections. Gold annotations are augmented with ontology-grounded synonym sets (MONDO, NCBI Gene, and NCBI Taxonomy) to support both lexical and concept-level evaluation. The dataset was constructed through a multi-stage human–AI pipeline combining structured human annotation, triage, iterative verification, and LLM-as-a-judge validation. To reduce leaderboard gaming and contamination risk, the 1000 scored questions are embedded within a publicly downloadable set of 10 000 questions, with answers withheld, on a CodaBench leaderboard. Evaluation of four frontier LLMs under a zero-shot setting (GPT-5.1, Gemini 2.5 Pro, Claude Sonnet 4.5, and GPT-4o) reveals performance variation across answer types, with overall accuracy ranging from 66.3% to 83.4%. Performance is strongest on chemical and anatomical questions and most variable on disease and gene/protein categories, where fine-grained semantic discrimination is required. MedHopQA provides both a benchmark and a reusable framework for constructing future biomedical QA datasets that prioritize compositional reasoning, saturation resistance, and contamination mitigation as design constraints.
- 15 January 2026
- Published by Oxford University Press (OUP) in Database: The Journal of Biological Databases and Curation
- Vol. 2026
Open AccessMedHopQA: a disease-centred multi-hop reasoning benchmark and evaluation framework for LLM-based biomedical question answering
Evaluating large language models (LLMs) in the biomedical domain requires benchmarks that can distinguish genuine reasoning from pattern matching and that remain discriminative as model capabilities improve. Existing biomedical question answering (QA) benchmarks are limited in this respect: multiple-choice formats allow models to succeed by answer elimination rather than inference, and widely circulated exam-style datasets are subject to performance saturation and training data contamination. Multi-hop reasoning, i.e. the ability to integrate information across multiple sources to derive an answer, is central to clinically meaningful tasks such as diagnosis support, literature-based discovery, and hypothesis generation, yet remains underrepresented in current biomedical QA benchmarks. We present MedHopQA, a disease-centred multi-hop reasoning benchmark of 1000 expert-curated question–answer pairs introduced as a shared task at BioCreative IX. Each question requires synthesis of information across two distinct Wikipedia articles, and answers are provided in open-ended free-text format rather than as multiple-choice selections. Gold annotations are augmented with ontology-grounded synonym sets (MONDO, NCBI Gene, and NCBI Taxonomy) to support both lexical and concept-level evaluation. The dataset was constructed through a multi-stage human–AI pipeline combining structured human annotation, triage, iterative verification, and LLM-as-a-judge validation. To reduce leaderboard gaming and contamination risk, the 1000 scored questions are embedded within a publicly downloadable set of 10 000 questions, with answers withheld, on a CodaBench leaderboard. Evaluation of four frontier LLMs under a zero-shot setting (GPT-5.1, Gemini 2.5 Pro, Claude Sonnet 4.5, and GPT-4o) reveals performance variation across answer types, with overall accuracy ranging from 66.3% to 83.4%. Performance is strongest on chemical and anatomical questions and most variable on disease and gene/protein categories, where fine-grained semantic discrimination is required. MedHopQA provides both a benchmark and a reusable framework for constructing future biomedical QA datasets that prioritize compositional reasoning, saturation resistance, and contamination mitigation as design constraints.
- Open Access
- 1 July 2025
- Published by Oxford University Press (OUP) in Bioinformatics
- Vol. 41 (Supplement) , i68-i76
Enhancing biomedical relation extraction with directionality
Summary: Biological relation networks contain rich information for understanding the biological mechanisms behind the relationship of entities such as genes, proteins, diseases, and chemicals. The vast growth of biomedical literature poses significant challenges in updating the network knowledge. The recent Biomedical Relation Extraction Dataset (BioRED) provides valuable manual annotations, facilitating the development of machine learning and pre-trained language model approaches for automatically identifying novel document-level (inter-sentence context) relationships. Nonetheless, its annotations lack directionality (subject/object) for the entity roles, which is essential for studying complex biological networks. Herein, we annotate the entity roles of the relationships in the BioRED corpus and subsequently propose a novel multi-task language model with soft-prompt learning to jointly identify the relationship, novel findings, and entity roles. Our results include an enriched BioRED corpus with 10 864 directionality annotations. Moreover, our proposed method outperforms existing large language models, such as the state-of-the-art GPT-4 and Llama-3, on two benchmarking tasks. Availability and implementation: Our source code and dataset are available at https://github.com/ncbi-nlp/BioREDirect.
- 1 July 2025
- Published by Oxford University Press (OUP) in Bioinformatics
- Vol. 41 (Supplement) , i68-i76
Open AccessEnhancing biomedical relation extraction with directionality
Summary: Biological relation networks contain rich information for understanding the biological mechanisms behind the relationship of entities such as genes, proteins, diseases, and chemicals. The vast growth of biomedical literature poses significant challenges in updating the network knowledge. The recent Biomedical Relation Extraction Dataset (BioRED) provides valuable manual annotations, facilitating the development of machine learning and pre-trained language model approaches for automatically identifying novel document-level (inter-sentence context) relationships. Nonetheless, its annotations lack directionality (subject/object) for the entity roles, which is essential for studying complex biological networks. Herein, we annotate the entity roles of the relationships in the BioRED corpus and subsequently propose a novel multi-task language model with soft-prompt learning to jointly identify the relationship, novel findings, and entity roles. Our results include an enriched BioRED corpus with 10 864 directionality annotations. Moreover, our proposed method outperforms existing large language models, such as the state-of-the-art GPT-4 and Llama-3, on two benchmarking tasks. Availability and implementation: Our source code and dataset are available at https://github.com/ncbi-nlp/BioREDirect.
- Open Access
- 9 September 2024
- Published by Springer Nature in Scientific Data
- Vol. 11 (1) , 1-19
EnzChemRED, a rich enzyme chemistry relation extraction dataset
Expert curation is essential to capture knowledge of enzyme functions from the scientific literature in FAIR open knowledgebases but cannot keep pace with the rate of new discoveries and new publications. In this work we present EnzChemRED, for Enzyme Chemistry Relation Extraction Dataset, a new training and benchmarking dataset to support the development of Natural Language Processing (NLP) methods such as (large) language models that can assist enzyme curation. EnzChemRED consists of 1,210 expert curated PubMed abstracts where enzymes and the chemical reactions they catalyze are annotated using identifiers from the protein knowledgebase UniProtKB and the chemical ontology ChEBI. We show that fine-tuning language models with EnzChemRED significantly boosts their ability to identify proteins and chemicals in text (86.30% F1 score) and to extract the chemical conversions (86.66% F1 score) and the enzymes that catalyze those conversions (83.79% F1 score). We apply our methods to abstracts at PubMed scale to create a draft map of enzyme functions in literature to guide curation efforts in UniProtKB and the reaction knowledgebase Rhea.
- 9 September 2024
- Published by Springer Nature in Scientific Data
- Vol. 11 (1) , 1-19
Open AccessEnzChemRED, a rich enzyme chemistry relation extraction dataset
Expert curation is essential to capture knowledge of enzyme functions from the scientific literature in FAIR open knowledgebases but cannot keep pace with the rate of new discoveries and new publications. In this work we present EnzChemRED, for Enzyme Chemistry Relation Extraction Dataset, a new training and benchmarking dataset to support the development of Natural Language Processing (NLP) methods such as (large) language models that can assist enzyme curation. EnzChemRED consists of 1,210 expert curated PubMed abstracts where enzymes and the chemical reactions they catalyze are annotated using identifiers from the protein knowledgebase UniProtKB and the chemical ontology ChEBI. We show that fine-tuning language models with EnzChemRED significantly boosts their ability to identify proteins and chemicals in text (86.30% F1 score) and to extract the chemical conversions (86.66% F1 score) and the enzymes that catalyze those conversions (83.79% F1 score). We apply our methods to abstracts at PubMed scale to create a draft map of enzyme functions in literature to guide curation efforts in UniProtKB and the reaction knowledgebase Rhea.
- Top Cited PapersOpen Access
- 4 April 2024
- Published by Oxford University Press (OUP) in Nucleic Acids Research
- Vol. 52 (W1) , W540-W546
PubTator 3.0: an AI-powered literature resource for unlocking biomedical knowledge
PubTator 3.0 (https://www.ncbi.nlm.nih.gov/research/pubtator3/) is a biomedical literature resource using state-of-the-art AI techniques to offer semantic and relation searches for key concepts like proteins, genetic variants, diseases and chemicals. It currently provides over one billion entity and relation annotations across approximately 36 million PubMed abstracts and 6 million full-text articles from the PMC open access subset, updated weekly. PubTator 3.0's online interface and API utilize these precomputed entity relations and synonyms to provide advanced search capabilities and enable large-scale analyses, streamlining many complex information needs. We showcase the retrieval quality of PubTator 3.0 using a series of entity pair queries, demonstrating that PubTator 3.0 retrieves a greater number of articles than either PubMed or Google Scholar, with higher precision in the top 20 results. We further show that integrating ChatGPT (GPT-4) with PubTator APIs dramatically improves the factuality and verifiability of its responses. In summary, PubTator 3.0 offers a comprehensive set of features and tools that allow researchers to navigate the ever-expanding wealth of biomedical literature, expediting research and unlocking valuable insights for scientific discovery.
- 4 April 2024
- Published by Oxford University Press (OUP) in Nucleic Acids Research
- Vol. 52 (W1) , W540-W546
Top Cited PapersOpen AccessPubTator 3.0: an AI-powered literature resource for unlocking biomedical knowledge
PubTator 3.0 (https://www.ncbi.nlm.nih.gov/research/pubtator3/) is a biomedical literature resource using state-of-the-art AI techniques to offer semantic and relation searches for key concepts like proteins, genetic variants, diseases and chemicals. It currently provides over one billion entity and relation annotations across approximately 36 million PubMed abstracts and 6 million full-text articles from the PMC open access subset, updated weekly. PubTator 3.0's online interface and API utilize these precomputed entity relations and synonyms to provide advanced search capabilities and enable large-scale analyses, streamlining many complex information needs. We showcase the retrieval quality of PubTator 3.0 using a series of entity pair queries, demonstrating that PubTator 3.0 retrieves a greater number of articles than either PubMed or Google Scholar, with higher precision in the top 20 results. We further show that integrating ChatGPT (GPT-4) with PubTator APIs dramatically improves the factuality and verifiability of its responses. In summary, PubTator 3.0 offers a comprehensive set of features and tools that allow researchers to navigate the ever-expanding wealth of biomedical literature, expediting research and unlocking valuable insights for scientific discovery.
- Top Cited Papers
- 31 May 2023
- Published by Wolters Kluwer Health in Journal of the American Society of Nephrology
- Vol. 34 (8) , 1302-1304
Retrieve, Summarize, and Verify: How Will ChatGPT Affect Information Seeking from the Medical Literature?
An abstract is unavailable. This article is available as a PDF only.
- 31 May 2023
- Published by Wolters Kluwer Health in Journal of the American Society of Nephrology
- Vol. 34 (8) , 1302-1304
Top Cited PapersRetrieve, Summarize, and Verify: How Will ChatGPT Affect Information Seeking from the Medical Literature?
An abstract is unavailable. This article is available as a PDF only.