Addressing cognitive bias in medical language models
Preprint
Abstract
There is increasing interest in the application large language models (LLMs) to the medical field, in part because of their impressive performance on medical exam questions. While promising, exam questions do not reflect the complexity of real patient-doctor interactions. In reality, physicians' decisions are shaped by many complex factors, such as patient compliance, personal experience, ethical beliefs, and cognitive bias. Taking a step toward understanding this, our hypothesis posits that when LLMs are confronted with clinical questions containing cognitive biases, they will yield significantly less accurate responses compared to the same questions presented without such biases. In this study, we developed BiasMedQA, a benchmark for evaluating cognitive biases in LLMs applied to medical tasks. Using BiasMedQA we evaluated six LLMs, namely GPT-4, Mixtral-8x70B, GPT-3.5, PaLM-2, Llama 2 70B-chat, and the medically specialized PMC Llama 13B. We tested these models on 1,273 questions from the US Medical Licensing Exam (USMLE) Steps 1, 2, and 3, modified to replicate common clinically-relevant cognitive biases. Our analysis revealed varying effects for biases on these LLMs, with GPT-4 standing out for its resilience to bias, in contrast to Llama 2 70B-chat and PMC Llama 13B, which were disproportionately affected by cognitive bias. Our findings highlight the critical need for bias mitigation in the development of medical LLMs, pointing towards safer and more reliable applications in healthcare.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Toward expert-level medical question answering with large language modelsNature Medicine, 2025
- The potential and pitfalls of using a large language model such as ChatGPT, GPT-4, or LLaMA as a clinical assistantJournal of the American Medical Informatics Association, 2024
- PMC-LLaMA: toward building open-source language models for medicineJournal of the American Medical Informatics Association, 2024
- Doctor Versus Artificial Intelligence: Patient and Physician Evaluation of Large Language Model Responses to Rheumatology Patient Questions in a Cross‐Sectional StudyArthritis & Rheumatology, 2024
- Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation studyThe Lancet Digital Health, 2023
- Large language models propagate race-based medicinenpj Digital Medicine, 2023
- Large language models encode clinical knowledgeNature, 2023
- Embracing Large Language Models for Medical Applications: Opportunities and ChallengesCureus, 2023
- Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicineEBioMedicine, 2023
- What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical ExamsApplied Sciences, 2021