Systematic benchmarking demonstrates large language models have not reached the diagnostic accuracy of traditional rare-disease decision support tools
Authors:
- Justin T ReeseJRJustin T Reese
- Leonardo ChimirriLCLeonardo Chimirri
- Yasemin BridgesYBYasemin Bridges
- Daniel DanisDDDaniel Danis
- J Harry CaufieldJCJ Harry Caufield
- Michael A. GarganoMGMichael A. Gargano
- Carlo KrollCKCarlo Kroll
- Andrew SchmederASAndrew Schmeder
- Fengchen LiuFLFengchen Liu
- Kyran WissinkKWKyran Wissink
- Julie A McMurryJMJulie A McMurry
- Adam Sl GraefeAGAdam Sl Graefe
Preprint
Abstract
Large language models (LLMs) show promise in supporting differential diagnosis, but their performance is challenging to evaluate due to the unstructured nature of their responses and their accuracy compared to existing diagnostic tools is not well characterized. To assess the current capabilities of LLMs to diagnose genetic diseases, we benchmarked these models on 5,213 case reports using the Phenopacket Schema, the Human Phenotype Ontology and Mondo disease ontology. Prompts generated from each phenopacket were sent to seven LLMs, including four generalist models and three LLMs specialized for medical applications. The same phenopackets were used as input to a widely used diagnostic tool, Exomiser, in phenotype-only mode. The best LLM ranked the correct diagnosis first in 23.6% of cases, whereas Exomiser did so in 35.5% of cases. While the performance of LLMs for supporting differential diagnosis has been improving, it has not reached the level of commonly used traditional bioinformatics tools. Future research is needed to determine the best approach to incorporate LLMs into diagnostic pipelines.Keywords
This publication has 37 references indexed in Scilit:
- A corpus of GA4GH Phenopackets: case-level phenotyping for genomic diagnostics and discoverymedRxiv, 2024
- On the limitations of large language models in clinical diagnosismedRxiv, 2023
- Large language models encode clinical knowledgeNature, 2023
- Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic ChallengeJAMA, 2023
- Foundation models for generalist medical artificial intelligenceNature, 2023
- The GA4GH Phenopacket schema defines a computable representation of clinical dataNature Biotechnology, 2022
- Case 16-2021: A 37-Year-Old Woman with Abdominal Pain and Aortic DilatationNew England Journal of Medicine, 2021
- Case 2-2021: A 26-Year-Old Pregnant Woman with Ventricular Tachycardia and ShockNew England Journal of Medicine, 2021
- The Monarch Initiative in 2019: an integrative data and analytic platform connecting phenotypes to genotypes across speciesNucleic Acids Research, 2019
- How many rare diseases are there?Nature Reviews Drug Discovery, 2019