Amagoia Madina
Research Subjects
Amagoia Madina has a total of 21 co-authors. Here are the top co-authors by number of shared publications:
- Ana Catarina de Sousa Teixeira2 shared publications
- Athina Stravodimou
Lausanne University Hospital 4 Medical Oncology Service, Department of Oncology Lausanne, Switzerland
2 shared publications - Bogdan Kulynych
Lausanne University Hospital (CHUV), Lausanne, Switzerland
2 shared publications - Carlos Fidalgo2 shared publications
- Christian Wandall Thorball
Lausanne University Hospital and University of Lausanne, Rue du Bugnon 46, 1005, Lausanne, Switzerland
2 shared publications - Coralie Galland-Decker
Department of Medicine, Internal Medicine, Lausanne University Hospital and University of Lausanne, Lausanne, Switzerland
2 shared publications
Latest Publications
- Open Access
- 4 November 2025
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 28, e86453
“Small” Large Language Models in the Hospital: An Evaluation Study on Real-World Data in a Resource-Constrained Setting (Preprint)
Background: Large language models (LLMs) are increasingly being deployed in health care, but their use and deployment in many real-world hospital environments pose significant challenges and concerns. In particular, state-of-the-art commercial models store or process data externally, which is often in conflict with ensuring patient data protection. At the same time, using LLMs locally is limited by the lack of available computing infrastructure. Small open-source LLMs that do not require substantial computing resources could offer a practical way to resolve these tensions, but their medical utility in real-world local contexts, especially in non-English languages, has not been sufficiently evaluated. Objective: This study aimed to evaluate the feasibility of small, locally deployable open-source LLMs for clinically relevant tasks in a resource-constrained hospital setting and to propose a reproducible framework for institution-specific evaluation before deployment. Methods: We evaluated 6 open-source LLMs ranging from 8B to 24B parameters (from the Mistral, Phi4, Falcon3, Llama3.1, and Meditron3 families) in a zero-shot setting across 7 tasks covering 4 clinical use cases: information extraction, medical text translation, text generation, and clinical decision support. We used deidentified French clinical data from a Swiss tertiary hospital, including discharge letters, clinical notes, and structured electronic health records. Performance was assessed using task-specific metrics, such as precision, recall, F1-score, embedding-based semantic similarity, recall-oriented understudy for gisting evaluation (ROUGE) score, readability indices, and human review by clinicians. Results: Model performance varied substantially between tasks. In the simplest retrieval task (needle-in-the-haystack), several models performed strongly, with Llama3.1 achieving an F1-score of 99.81% and Mistral-small achieving 99.71%. In contrast, performance was poor in more complex tasks. For detecting protected health information, the best-performing LLMs achieved only modest overall macro–F1-scores (0.33-0.34), substantially below a fine-tuned Robustly Optimized BERT Pretraining Approach (RoBERTa) baseline (0.94). In the task of extracting immune-related adverse events from discharge notes, the highest overall macro–F1-score was 0.35 with Phi4. For medical text translation, Phi4 ranked highest in embedding-based evaluation, whereas Meditron3-Phi4 performed the worst, with clinician reviews identifying hallucinations in 55% of its outputs. In the task of summarizing discharge letters, quality was low across all models, with the best penalized ROUGE score reaching only 0.169 with Llama3.1. In the tasks of generating patient-friendly discharge note summaries and clinical decision support, clinician ratings generally ranged from dissatisfied to neutral, and no model achieved consistently satisfactory performance. Conclusions: Small open-source LLMs appear feasible for simple retrieval-oriented tasks in local hospital deployments but are currently inadequate for more complex applications, such as clinical decision support, deidentification, extraction of adverse events, and medical summarization. These findings highlight the importance of locally grounded evaluation tailored to specific use cases and the need for robust institutional evaluation frameworks to ensure safe and reliable deployment.
- 4 November 2025
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 28, e86453
Open Access“Small” Large Language Models in the Hospital: An Evaluation Study on Real-World Data in a Resource-Constrained Setting (Preprint)
Background: Large language models (LLMs) are increasingly being deployed in health care, but their use and deployment in many real-world hospital environments pose significant challenges and concerns. In particular, state-of-the-art commercial models store or process data externally, which is often in conflict with ensuring patient data protection. At the same time, using LLMs locally is limited by the lack of available computing infrastructure. Small open-source LLMs that do not require substantial computing resources could offer a practical way to resolve these tensions, but their medical utility in real-world local contexts, especially in non-English languages, has not been sufficiently evaluated. Objective: This study aimed to evaluate the feasibility of small, locally deployable open-source LLMs for clinically relevant tasks in a resource-constrained hospital setting and to propose a reproducible framework for institution-specific evaluation before deployment. Methods: We evaluated 6 open-source LLMs ranging from 8B to 24B parameters (from the Mistral, Phi4, Falcon3, Llama3.1, and Meditron3 families) in a zero-shot setting across 7 tasks covering 4 clinical use cases: information extraction, medical text translation, text generation, and clinical decision support. We used deidentified French clinical data from a Swiss tertiary hospital, including discharge letters, clinical notes, and structured electronic health records. Performance was assessed using task-specific metrics, such as precision, recall, F1-score, embedding-based semantic similarity, recall-oriented understudy for gisting evaluation (ROUGE) score, readability indices, and human review by clinicians. Results: Model performance varied substantially between tasks. In the simplest retrieval task (needle-in-the-haystack), several models performed strongly, with Llama3.1 achieving an F1-score of 99.81% and Mistral-small achieving 99.71%. In contrast, performance was poor in more complex tasks. For detecting protected health information, the best-performing LLMs achieved only modest overall macro–F1-scores (0.33-0.34), substantially below a fine-tuned Robustly Optimized BERT Pretraining Approach (RoBERTa) baseline (0.94). In the task of extracting immune-related adverse events from discharge notes, the highest overall macro–F1-score was 0.35 with Phi4. For medical text translation, Phi4 ranked highest in embedding-based evaluation, whereas Meditron3-Phi4 performed the worst, with clinician reviews identifying hallucinations in 55% of its outputs. In the task of summarizing discharge letters, quality was low across all models, with the best penalized ROUGE score reaching only 0.169 with Llama3.1. In the tasks of generating patient-friendly discharge note summaries and clinical decision support, clinician ratings generally ranged from dissatisfied to neutral, and no model achieved consistently satisfactory performance. Conclusions: Small open-source LLMs appear feasible for simple retrieval-oriented tasks in local hospital deployments but are currently inadequate for more complex applications, such as clinical decision support, deidentification, extraction of adverse events, and medical summarization. These findings highlight the importance of locally grounded evaluation tailored to specific use cases and the need for robust institutional evaluation frameworks to ensure safe and reliable deployment.
- Open Access
- 4 November 2025
- Published by JMIR Publications Inc. in JMIR Preprints
“Small” Large Language Models in the hospital: an evaluation study on real-world data in a resources-constrained setting (Preprint)
BACKGROUND Large Language Models (LLMs) offer promise for healthcare but face challenges of scale, privacy, and limited evidence in non-English settings. Smaller, locally deployable LLMs remain underexplored. OBJECTIVE To assess the feasibility of small open-source LLMs (1–24B parameters) in French-language clinical tasks and provide a reproducible hospital-based evaluation framework. METHODS Six state-of-the-art small LLMs from the Mistral, Phi-4, Llama-3.1, Meditron-3, Falcon 3 model families were tested in a zero-shot setting on de-identified discharge letters across seven use cases, including information extraction, translation, summarization, and clinical decision support. Performance was measured with F1 scores, readability indices, embedding similarity, and structured clinician reviews. RESULTS The models achieved high recall in simple retrieval tasks (up to 99.6%) but showed poor performance in detection of protected health information, adverse-event extraction, summarization, and decision support. Translation quality varied, with general-purpose models outperforming medical-focused models. CONCLUSIONS In localized resource-constrained deployments, small LLMs are suitable for basic tasks but insufficient for complex reasoning or clinical decision-making. Our framework supports context-specific evaluation for safe adoption in hospitals.
- 4 November 2025
- Published by JMIR Publications Inc. in JMIR Preprints
Open Access“Small” Large Language Models in the hospital: an evaluation study on real-world data in a resources-constrained setting (Preprint)
BACKGROUND Large Language Models (LLMs) offer promise for healthcare but face challenges of scale, privacy, and limited evidence in non-English settings. Smaller, locally deployable LLMs remain underexplored. OBJECTIVE To assess the feasibility of small open-source LLMs (1–24B parameters) in French-language clinical tasks and provide a reproducible hospital-based evaluation framework. METHODS Six state-of-the-art small LLMs from the Mistral, Phi-4, Llama-3.1, Meditron-3, Falcon 3 model families were tested in a zero-shot setting on de-identified discharge letters across seven use cases, including information extraction, translation, summarization, and clinical decision support. Performance was measured with F1 scores, readability indices, embedding similarity, and structured clinician reviews. RESULTS The models achieved high recall in simple retrieval tasks (up to 99.6%) but showed poor performance in detection of protected health information, adverse-event extraction, summarization, and decision support. Translation quality varied, with general-purpose models outperforming medical-focused models. CONCLUSIONS In localized resource-constrained deployments, small LLMs are suitable for basic tasks but insufficient for complex reasoning or clinical decision-making. Our framework supports context-specific evaluation for safe adoption in hospitals.