Human-Machine Agreement in Medical Ethics: Patient Autonomy Case-Based Evaluation of Large Language Models
Open Access
- 23 October 2025
- journal article
- Published by JMIR Publications Inc. in JMIR Medical Informatics
- Vol. 13, e77061
Abstract
Background: Medical ethics provides a moral framework for the practice of clinical medicine. Four principles, that is, beneficence, nonmaleficence, patient autonomy, and justice, form the cornerstones of medical ethics as it is practiced today. Of these 4 principles, patient autonomy holds a pivotal position and often takes precedence in ethical dilemmas that result from conflicts among the 4 principles. Its importance serves as a constant reminder to the clinician that the “needs of the patient come first.” With their remarkable ability to process natural language, large language models (LLMs) have recently pervaded nearly every aspect of human life, including medicine and medical ethics. Reliance on tools such as LLMs, however, poses fundamental questions in medical ethics, where human-like reasoning, emotional intelligence, and an understanding of local context and values are of utmost importance. Objective: While emphasizing the central role of the human factor, we undertake a bold venture to establish some confidence in LLMs, as it pertains to medical ethics by not only evaluating the status quo of foundational LLMs but also exploring ways to improve the LLMs by using patient autonomy–based hypothetical cases. Although literature today is certainly lacking in such ventures, we also believe projects such as ours must be frequently revisited in the field of LLMs, which is evolving at a pace that is both rapid and unprecedented. Methods: We evaluated 3 foundational LLMs (ChatGPT, LLaMA, and Gemini) on hypothetical cases in patient autonomy. We used Cohen κ to compare LLM responses to the consensus from a physician panel. McNemar test was used during the improvement phase and to report the final significance of improved agreement of each LLM with physician consensus. P values less than .05 were considered significant. An agreement with κ<0 was designated as poor, 0-0.2 as slight, 0.2-0.4 as fair, 0.41-0.6 as moderate, 0.61-0.8 as substantial, and 0.81-1 as almost perfect. Results: There was slight to fair agreement between the foundational LLMs and the physician consensus. With iterative improvement techniques, this agreement evolved to be substantial or higher (Cohen κ of 0.73-0.82). The degree of improvement was statistically significant (P=.006 for ChatGPT, P<.001 for Gemini, and P<.001 for LLaMA). Conclusions: Although LLMs hold great potential for use in medicine, there needs to be an abundance of caution in using foundational LLMs in domains such as medical ethics. With adequate human oversight in testing and utilizing established techniques, LLM responses can be better aligned to human responses, even in the domain of medical ethics.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- The long but necessary road to responsible use of large language models in healthcare researchnpj Digital Medicine, 2024
- The new paradigm in machine learning – foundation models, large language models and beyond: a primer for physiciansInternal Medicine Journal, 2024
- A Personalized Patient Preference Predictor for Substituted Judgments in Healthcare: Technically Feasible and Ethically DesirableThe American Journal of Bioethics, 2024
- Exploring the potential utility of AI large language models for medical ethics: an expert panel evaluation of GPT-4Journal of Medical Ethics, 2023
- GPT-4 Artificial Intelligence Model Outperforms ChatGPT, Medical Students, and Neurosurgery Residents on Neurosurgery Written Board-Like QuestionsWorld Neurosurgery, 2023
- Evaluating the Feasibility of ChatGPT in Healthcare: An Analysis of Multiple Clinical and Research ScenariosJournal of Medical Systems, 2023
- Respecter l’autonomie de la personne soignéeSoins, 2018
- Respecting Patient Autonomy Versus Protecting the Patient's HealthJONA's Healthcare Law, Ethics, and Regulation, 2009
- The Principlism Debate: A Critical OverviewThe Journal of Medicine and Philosophy: A Forum for Bioethics and Philosophy of Medicine, 1995
- A Critique of PrinciplismThe Journal of Medicine and Philosophy: A Forum for Bioethics and Philosophy of Medicine, 1990