Implicit Bias in Large Language Model Diagnosis of Eating Disorders: Experimental Vignette Study
Open Access
- 23 September 2026
- journal article
- Published by JMIR Publications Inc. in JMIR AI
- Vol. 5, e93498
Abstract
Background: Large language models (LLMs) are increasingly deployed in mental health applications, yet growing evidence suggests they encode algorithmic biases that influence clinical outputs. Because these models now mediate patient-facing decisions, such biases carry the potential for direct harm. Whether they systematically affect psychiatric diagnosis across demographic groups remains underexplored. Objective: This study aims to examine whether LLMs exhibit implicit demographic biases when generating psychiatric diagnoses. Methods: We developed 1152 synthetic clinical vignettes using a matched-pair design that manipulated gender, race and ethnicity, age, socioeconomic status, English proficiency, and urbanicity while holding clinical content constant. Vignettes were divided into control (unambiguous anorexia nervosa [AN]) and ambiguous conditions designed to permit differential diagnosis. Ten LLM configurations across 5 model families were tested. Results: Control vignettes produced near-unanimous AN diagnoses (mean 100%, SD 0.1%), while ambiguous vignettes elicited greater variability (mean 23.6%, SD 10.1%). Intermodel agreement was moderate for ambiguous vignettes (Fleiss κ=0.410, 95% CI 0.397‐0.422). Mixed-effects logistic regression with LLM as a random intercept revealed significant demographic biases: Black patients were over 6 times more likely to receive a major depressive disorder (MDD) diagnosis than White patients with identical presentations (odds ratio [OR] 6.09, 95% CI 5.13‐7.24), Latine patients were over 9 times more likely (OR 9.57, 95% CI 8.00‐11.45), and Asian patients were nearly 3 times more likely to receive an AN diagnosis (OR 2.88, 95% CI 2.44‐3.42). Female patients were less likely than males to be diagnosed with AN (OR 0.43, 95% CI 0.37‐0.49). Conclusions: These findings demonstrate that LLMs exhibit systematic demographic biases in psychiatric diagnosis even when clinical content is held constant, revealing measurable patterns that can inform improvements to training data, model architecture, and clinical deployment frameworks.Keywords
This publication has 34 references indexed in Scilit:
- Eating Disorder Risk and Diagnosis Among East Asian Youth in the United States: Findings From the Healthy Minds Study, 2020–2023International Journal of Eating Disorders, 2025
- Racial bias in AI-mediated psychiatric diagnosis and treatment: a qualitative comparison of four large language modelsnpj Digital Medicine, 2025
- Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation studyThe Lancet Digital Health, 2023
- Large language models in medicineNature Medicine, 2023
- Eating disorder symptoms in Asian American college studentsEating Behaviors, 2020
- Challenges and barriers in mental healthcare systems and their impact on the family: A systematic integrative reviewHealth & Social Care in the Community, 2020
- Dissecting racial bias in an algorithm used to manage the health of populationsScience, 2019
- Disparities in eating disorder diagnosis and treatment according to weight status, race/ethnicity, socioeconomic background, and sex among college studentsInternational Journal of Eating Disorders, 2018
- Semantics derived automatically from language corpora contain human-like biasesScience, 2017
- Differences in the Expression of Symptoms in Men Versus Women with Depression: A Systematic Review and Meta-analysisHarvard Review of Psychiatry, 2017