Large Language Model–Generated Patient Instructions for Prescriptions in Primary Health Care: Preclinical Algorithm Validation (Preprint)
Authors:
- Zilma Silveira Nogueira ReisZRZilma Silveira Nogueira Reis
- Elisa Tuler AlbergariaEAElisa Tuler Albergaria
- Adriana Silvina PaganoAPAdriana Silvina Pagano
- Eura Martins LageELEura Martins Lage
- Flávia Ribeiro de OliveiraFOFlávia Ribeiro de Oliveira
- Cristiane dos Santos DiasCDCristiane dos Santos Dias
- Juliana Almeida OliveiraJOJuliana Almeida Oliveira
- Gláucia Miranda Varella PereiraGPGláucia Miranda Varella Pereira
- Isaias Jose Ramos de OliveiraIOIsaias Jose Ramos de Oliveira
- Érico Franco MineiroÉMÉrico Franco Mineiro
- Igor Carvalho Lima OliveiraIOIgor Carvalho Lima Oliveira
- Davi dos Reis de JesusDJDavi dos Reis de Jesus
Preprint
- 19 September 2025
- preprint
- Published by JMIR Publications Inc. in JMIR Preprints
Abstract
BACKGROUND The application of generative artificial intelligence to simplify medication use instructions has the potential to enhance people’s health by improving treatment adherence. OBJECTIVE We evaluated the performance of large language models (LLMs) in generating medication usage instructions to complement prescriptions in primary health care. METHODS This randomized, blinded experimental preclinical study used prescription-inducing scenarios, assigned to 62 health care professionals, to validate instructions generated by LLMs during electronic prescriptions. The instructions were generated by ChatGPT-4.0 (OpenAI), Llama3.1-8B (Meta), and Llama3.1-8B-RAG (Meta) using retrieval-augmented generation based on patient information leaflets. Performance metrics assessed adequacy, completeness, clarity, language simplification, usefulness, and errors in the generated instructions, with scores to analyze overall and individual metrics. RESULTS The 3 models yielded high overall scores for producing qualified instructions (ChatGPT-4.0: median 88.4, IQR 22.8; Llama3.1-8B: median 66.5, IQR 50.9; Llama3.1-8B-RAG: median 79.9, IQR 34.4; Kruskal-Wallis test P=.003). Llama3.1-8B-RAG received evaluations with similar overall scores to ChatGPT-4.0 (post hoc test, P=.05) and similar to Llama3.1-8B (post hoc test, P=.44). ChatGPT-4.0 outperformed Llama3.1-8B (Bonferroni test, P<.001). Regarding specific domains, Llama3.1-8B-RAG received scores equivalent to those of ChatGPT-4.0 for adequacy (mean 6.24, SD 2.3 vs mean 6.82, SD 2.1; post hoc test, P=.54); completeness (mean 5.94, SD 2.2 vs 6.55, SD 1.9; post hoc test P=.38), clarity (mean 5.77, SD 2.4 vs mean 6.68, SD 1.9; post hoc test P=.09), and usefulness (mean 5.42, SD 2.4 vs mean 5.96, SD 2.2; post hoc test P=.63). ChatGPT-4.0 received higher scores in the language simplification criterion than Llama3.1-8B-RAG (mean 7.05, SD 1.5 vs mean 5.44, SD 2.6; post hoc test P<.001). Interrater variability in assigning scores ranged from 4.2% (n=3) to 85.8% (n=6) among primary health care professionals. Instructions leading to incorrect use of the medication had similar frequency among the models(ChatGPT-4.0: n=15, 22.7%; Llama3.1-8B: n=19, 22.8%; Llama3.1-8B-RAG: n=19, 22.8%; chi-square test P=.71). The frequencies of hallucination were similar (ChatGPT-4.0: n=7, 10.6%; Llama3.1-8B: n=9, 13.6%; Llama3.1-8B-RAG: n=6, 9.1%; chi-square test P=.67). CONCLUSIONS The open-source LLM enhanced with external information presented similar performance to the closed-source model, except for ChatGPT4.0, which was superior in language simplification of messages. LLM generation demonstrated potential for instructing patients on medication use. Nonetheless, the introduction of this innovation into the electronic prescribing workflow demands prescriber validation for human oversight of the technology and requires a strategy for LLM performance governance. INTERNATIONAL REGISTERED REPORT RR2-10.12688/verixiv.1359.1Keywords
All Related Versions
- Published version: Journal of Medical Internet Research, 28, e84444.
This publication has 7 references indexed in Scilit:
- Picobirnavirus: how do you find where it’s hiding?Critical Reviews in Microbiology, 2025
- Quantization-Based Jailbreaking Vulnerability Analysis: A Study on Performance and Safety of the Llama3-8B-Instruct ModelIEEE Access, 2025
- Declaration of Interests☒ the Authors Declare that They Have No Known Competing Financial Interests or Personal Relationships that Could Have Appeared to Influence the Work Reported in this Paper. ☐ the Authors Declare the Following Financial Interests/Personal Relationships Which May Be Considered as Potential Competing Interests:SSRN Electronic Journal, 2024
- Дистанційне навчання, Google calendar, Google Slides, Google Docs, Google Chat.2023
- Trichotomization with two cutoff values using Kruskal-Wallis test by minimum P-value approachJournal of Applied Mathematics, Statistics and Informatics, 2022
- Atuação da Agência Nacional de Vigilância Sanitária no enquadramento de produtos fronteira: uma pesquisa documentalVigilância Sanitária Em Debate: Sociedade, Ciência & Tecnologia, 2021
- Interquartile Range2011