“Small” Large Language Models in the Hospital: An Evaluation Study on Real-World Data in a Resource-Constrained Setting (Preprint)
Authors:
- He A. XuHXHe A. Xu
- Romain PythoudRPRomain Pythoud
- Christian W ThorballCTChristian W Thorball
- Giorgia CarraGCGiorgia Carra
- Bogdan KulynychBKBogdan Kulynych
- Jérémie DesprazJDJérémie Despraz
- Coralie Galland-DeckerCGCoralie Galland-Decker
- Errikos MasliasEMErrikos Maslias
- Edouard BaudsonEBEdouard Baudson
- Thomas BrahierTBThomas Brahier
- Vanessa KraegeVKVanessa Kraege
- Ana Catarina de Sousa TeixeiraATAna Catarina de Sousa Teixeira
Open Access
- 4 November 2025
- journal article
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 28, e86453
Abstract
Background: Large language models (LLMs) are increasingly being deployed in health care, but their use and deployment in many real-world hospital environments pose significant challenges and concerns. In particular, state-of-the-art commercial models store or process data externally, which is often in conflict with ensuring patient data protection. At the same time, using LLMs locally is limited by the lack of available computing infrastructure. Small open-source LLMs that do not require substantial computing resources could offer a practical way to resolve these tensions, but their medical utility in real-world local contexts, especially in non-English languages, has not been sufficiently evaluated. Objective: This study aimed to evaluate the feasibility of small, locally deployable open-source LLMs for clinically relevant tasks in a resource-constrained hospital setting and to propose a reproducible framework for institution-specific evaluation before deployment. Methods: We evaluated 6 open-source LLMs ranging from 8B to 24B parameters (from the Mistral, Phi4, Falcon3, Llama3.1, and Meditron3 families) in a zero-shot setting across 7 tasks covering 4 clinical use cases: information extraction, medical text translation, text generation, and clinical decision support. We used deidentified French clinical data from a Swiss tertiary hospital, including discharge letters, clinical notes, and structured electronic health records. Performance was assessed using task-specific metrics, such as precision, recall, F1-score, embedding-based semantic similarity, recall-oriented understudy for gisting evaluation (ROUGE) score, readability indices, and human review by clinicians. Results: Model performance varied substantially between tasks. In the simplest retrieval task (needle-in-the-haystack), several models performed strongly, with Llama3.1 achieving an F1-score of 99.81% and Mistral-small achieving 99.71%. In contrast, performance was poor in more complex tasks. For detecting protected health information, the best-performing LLMs achieved only modest overall macro–F1-scores (0.33-0.34), substantially below a fine-tuned Robustly Optimized BERT Pretraining Approach (RoBERTa) baseline (0.94). In the task of extracting immune-related adverse events from discharge notes, the highest overall macro–F1-score was 0.35 with Phi4. For medical text translation, Phi4 ranked highest in embedding-based evaluation, whereas Meditron3-Phi4 performed the worst, with clinician reviews identifying hallucinations in 55% of its outputs. In the task of summarizing discharge letters, quality was low across all models, with the best penalized ROUGE score reaching only 0.169 with Llama3.1. In the tasks of generating patient-friendly discharge note summaries and clinical decision support, clinician ratings generally ranged from dissatisfied to neutral, and no model achieved consistently satisfactory performance. Conclusions: Small open-source LLMs appear feasible for simple retrieval-oriented tasks in local hospital deployments but are currently inadequate for more complex applications, such as clinical decision support, deidentification, extraction of adverse events, and medical summarization. These findings highlight the importance of locally grounded evaluation tailored to specific use cases and the need for robust institutional evaluation frameworks to ensure safe and reliable deployment.Keywords
This publication has 38 references indexed in Scilit:
- The evaluation illusion of large language models in medicinenpj Digital Medicine, 2025
- Evaluating a Large Language Model in Translating Patient Instructions to Spanish Using a Standardized FrameworkJAMA Pediatrics, 2025
- Automated generation of discharge summaries: leveraging large language models with clinical dataScientific Reports, 2025
- The rise of agentic AI teammates in medicineThe Lancet, 2025
- LLM-based agentic systems in medicine and healthcareNature Machine Intelligence, 2024
- The future landscape of large language models in medicineCommunications Medicine, 2023
- Large language models encode clinical knowledgeNature, 2023
- Author Correction: MIMIC-IV, a freely accessible electronic health record datasetScientific Data, 2023
- MIMIC-III, a freely accessible critical care databaseScientific Data, 2016
- From The Field: The Politics Of The Health Insurance Portability And Accountability ActHealth Affairs, 1997