Addressing cognitive bias in medical language models
Preprint
Abstract
The integration of large language models (LLMs) into the medical field has gained significant attention due to their promising accuracy in simulated clinical decision-making settings. However, clinical decision-making is more complex than simulations because physicians' decisions are shaped by many factors, including the presence of cognitive bias. However, the degree to which LLMs are susceptible to the same cognitive biases that affect human clinicians remains unexplored. Our hypothesis posits that when LLMs are confronted with clinical questions containing cognitive biases, they will yield significantly less accurate responses compared to the same questions presented without such biases. In this study, we developed BiasMedQA, a novel benchmark for evaluating cognitive biases in LLMs applied to medical tasks. Using BiasMedQA we evaluated six LLMs, namely GPT-4, Mixtral-8x70B, GPT-3.5, PaLM-2, Llama 2 70B-chat, and the medically specialized PMC Llama 13B. We tested these models on 1,273 questions from the US Medical Licensing Exam (USMLE) Steps 1, 2, and 3, modified to replicate common clinically-relevant cognitive biases. Our analysis revealed varying effects for biases on these LLMs, with GPT-4 standing out for its resilience to bias, in contrast to Llama 2 70B-chat and PMC Llama 13B, which were disproportionately affected by cognitive bias. Our findings highlight the critical need for bias mitigation in the development of medical LLMs, pointing towards safer and more reliable applications in healthcare.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Toward expert-level medical question answering with large language modelsNature Medicine, 2025
- The potential and pitfalls of using a large language model such as ChatGPT, GPT-4, or LLaMA as a clinical assistantJournal of the American Medical Informatics Association, 2024
- PMC-LLaMA: toward building open-source language models for medicineJournal of the American Medical Informatics Association, 2024
- Doctor Versus Artificial Intelligence: Patient and Physician Evaluation of Large Language Model Responses to Rheumatology Patient Questions in a Cross‐Sectional StudyArthritis & Rheumatology, 2024
- Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation studyThe Lancet Digital Health, 2023
- Large language models propagate race-based medicinenpj Digital Medicine, 2023
- Large language models encode clinical knowledgeNature, 2023
- Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicineEBioMedicine, 2023
- What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical ExamsApplied Sciences, 2021
- Nursing Workload as a Risk Factor for Healthcare Associated Infections in ICU: A Prospective StudyPLOS ONE, 2012