MedBot vs RealDoc: efficacy of large language modeling in physician-patient communication for rare diseases
Open Access
- 25 February 2025
- journal article
- research article
- Published by Oxford University Press (OUP) in Journal of the American Medical Informatics Association
- Vol. 32 (5) , 775-783
Abstract
Objectives: This study assesses the abilities of 2 large language models (LLMs), GPT-4 and BioMistral 7B, in responding to patient queries, particularly concerning rare diseases, and compares their performance with that of physicians. Materials and Methods: A total of 103 patient queries and corresponding physician answers were extracted from EXABO, a question-answering forum dedicated to rare respiratory diseases. The responses provided by physicians and generated by LLMs were ranked on a Likert scale by a panel of 4 experts based on 4 key quality criteria for health communication: correctness, comprehensibility, relevance, and empathy. Results: The performance of generative pretrained transformer 4 (GPT-4) was significantly better than the performance of the physicians and BioMistral 7B. While the overall ranking considers GPT-4’s responses to be mostly correct, comprehensive, relevant, and emphatic, the responses provided by BioMistral 7B were only partially correct and empathetic. The responses given by physicians rank in between. The experts concur that an LLM could lighten the load for physicians, rigorous validation is considered essential to guarantee dependability and efficacy. Discussion: Open-source models such as BioMistral 7B offer the advantage of privacy by running locally in health-care settings. GPT-4, on the other hand, demonstrates proficiency in communication and knowledge depth. However, challenges persist, including the management of response variability, the balancing of comprehensibility with medical accuracy, and the assurance of consistent performance across different languages. Conclusion: The performance of GPT-4 underscores the potential of LLMs in facilitating physician-patient communication. However, it is imperative that these systems are handled with care, as erroneous responses have the potential to cause harm without the requisite validation procedures.Keywords
This publication has 27 references indexed in Scilit:
- The application of large language models in medicine: A scoping reviewiScience, 2024
- Large Language Model−Based Chatbot vs Surgeon-Generated Informed Consent Documentation for Common ProceduresJAMA Network Open, 2023
- Benchmarking large language models’ performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google BardEBioMedicine, 2023
- Comparison of Ophthalmologist and Large Language Model Chatbot Responses to Online Patient Eye Care QuestionsJAMA Network Open, 2023
- Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media ForumJAMA Internal Medicine, 2023
- Opinion Paper: “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policyInternational Journal of Information Management, 2023
- Digitization of healthcare sector: A study on privacy and security concernsICT Express, 2023
- Befundergebnisse verständlich vermitteln – Eine randomisiert-kontrollierte Studie zur Wortwahl in der Ärzt*innen-Patient*innen-KommunikationZeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen, 2020
- A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability ResearchJournal of Chiropractic Medicine, 2016
- What is an expert? A systems perspective on expertiseEcology and Evolution, 2013