How well do LLMs cite relevant medical references? An evaluation framework and analyses
Authors:
Preprint
Abstract
Large language models (LLMs) are currently being used to answer medical questions across a variety of clinical domains. Recent top-performing commercial LLMs, in particular, are also capable of citing sources to support their responses. In this paper, we ask: do the sources that LLMs generate actually support the claims that they make? To answer this, we propose three contributions. First, as expert medical annotations are an expensive and time-consuming bottleneck for scalable evaluation, we demonstrate that GPT-4 is highly accurate in validating source relevance, agreeing 88% of the time with a panel of medical doctors. Second, we develop an end-to-end, automated pipeline called \textit{SourceCheckup} and use it to evaluate five top-performing LLMs on a dataset of 1200 generated questions, totaling over 40K pairs of statements and sources. Interestingly, we find that between ~50% to 90% of LLM responses are not fully supported by the sources they provide. We also evaluate GPT-4 with retrieval augmented generation (RAG) and find that, even still, around 30\% of individual statements are unsupported, while nearly half of its responses are not fully supported. Third, we open-source our curated dataset of medical questions and expert annotations for future evaluations. Given the rapid pace of LLM development and the potential harms of incorrect or outdated medical information, it is crucial to also understand and quantify their capability to produce relevant, trustworthy medical references.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- Loneliness and suicide mitigation for students using GPT3-enabled chatbotsnpj Mental Health Research, 2024
- Towards Verifiable Generation: A Benchmark for Knowledge-aware Language Model Attribution2024
- ExpertQA: Expert-Curated Questions and Attributed Answers2024
- Large Legal Fictions: Profiling Legal Hallucinations in Large Language ModelsJournal of Legal Analysis, 2024
- How to Safely Integrate Large Language Models Into Health CareJAMA Health Forum, 2023
- Chatbot vs Medical Student Performance on Free-Response Clinical Reasoning ExaminationsJAMA Internal Medicine, 2023
- Large language model AI chatbots require approval as medical devicesNature Medicine, 2023
- Evaluating Verifiability in Generative Search Engines2023
- Enabling Large Language Models to Generate Text with Citations2023
- What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical ExamsApplied Sciences, 2021