Assessing GPT‐4's accuracy in answering clinical pharmacological questions on pain therapy
- 11 March 2025
- journal article
- Published by Wiley in British Journal of Clinical Pharmacology
- Vol. 91 (8) , 2294-2303
Abstract
Aims: This study aimed to evaluate the accuracy and completeness of GPT‐4, a large language model, in answering clinical pharmacological questions related to pain therapy, with a focus on its potential as a tool for delivering patient‐facing medical information. The objective was to assess its reliability in delivering medical information in the context of pain management.Methods: A cross‐sectional survey‐based study was conducted with healthcare professionals, including physicians and pharmacists. Participants submitted up to 8 clinical pharmacology questions on pain management, focusing on drug interactions, dosages and contraindications. GPT‐4's responses were evaluated based on comprehensibility, detail, satisfaction, medical–pharmacological accuracy and completeness. Additionally, responses were compared to the German Drug Directory to assess their accuracy.Results: The majority of participants (99%) found GPT‐4's responses comprehensible, while 84% considered the information detailed enough. Overall satisfaction was high, with 93% expressing satisfaction, and 96% deemed the responses medically accurate. However, only 63% rated the information as complete, with some identifying gaps in pharmacokinetics and drug interaction data. Usability was evaluated as good to excellent, with a System Usability Scale score of 83.38 (± 10.26).Conclusion: GPT‐4 demonstrates potential as a tool for delivering medical information, particularly in pain management. However, limitations such as incomplete pharmacological data and the potential for contextual carryover in follow‐up questions suggest that further refinement is necessary. Developing specialized artificial intelligence tools that integrate real‐time pharmacological databases could improve accuracy and reliability for clinical decision‐making.Keywords
This publication has 17 references indexed in Scilit:
- Testing and Evaluation of Health Care Applications of Large Language ModelsJAMA, 2025
- Open E-survey on the Use and Perception of Chatbots in Vascular SurgeryEJVES Vascular Forum, 2024
- Artificial intelligence and machine learning for clinical pharmacologyBritish Journal of Clinical Pharmacology, 2023
- Artificial Intelligence–Assisted Sac Diameter Assessment for Complex Endovascular Aortic RepairJournal of Endovascular Therapy, 2023
- An exploratory survey about using ChatGPT in education, healthcare, and researchPLOS ONE, 2023
- Chat GPT-4 significantly surpasses GPT-3.5 in drug information queriesJournal of Telemedicine and Telecare, 2023
- A Personal Resource for Technology Interaction: Development and Validation of the Affinity for Technology Interaction (ATI) ScaleInternational Journal of Human–Computer Interaction, 2018
- Association of Household Opioid Availability and Prescription Opioid Initiation Among Household MembersJAMA Internal Medicine, 2018
- Patient–provider interaction during medication encounters: A study in outpatient pharmacies in the NetherlandsPatient Education and Counseling, 2015
- COMPREHENSIBILITY OF THE PACKAGE LEAFLETS OF ALL MEDICINAL PRODUCTS FOR HUMAN USE: A QUESTIONNAIRE SURVEY ABOUT THE USE OF SYMBOLS AND PICTOGRAMSPharmacological Research, 2000