Ahmad Idrissi-Yaghir

This study evaluated a user-centered clinical dashboard using FHIR-Server data to improve information accessibility for physicians treating melanoma patients. Current electronic health record systems distribute critical information across multiple modules and prioritize billing and research over clinical decision-making. We employed a mixed-methods approach combining think-aloud protocols, qualitative content analysis, and standardized usability questionnaires across three iterative evaluation cycles with nine participants in eleven sessions. Our dashboard implementation demonstrated substantial workflow improvements, reducing required interactions from 17 to 23 clicks to just 5–6 clicks, enabling physicians to focus more on treatment decisions than information gathering. The Post-Study System Usability Questionnaire showed progressive enhancement across iterations, with overall scores improving from 2.4 to 1.8. Affinity diagrams revealed key improvements in information accessibility and workflow integration. These findings indicate the value of a user-centered, holistic approach to clinical dashboard development, showing that FHIR-based interoperability infrastructures can be transformed into point-of-care applications when information is structured around clinicians’ mental models rather than technically derived data structures. Beyond the melanoma context, this approach provides a transferable foundation for systematically exploring clinical contexts and translating these insights into meaningful, workflow-integrated applications across clinical domains.

Background: The standardized extraction of postoperative complications from unstructured routine clinical documentation remains a major unresolved challenge in digital surgery and health informatics. Although the Clavien-Dindo classification is the established standard for grading postoperative complications, its application in routine clinical documentation is largely implicit and unstructured, limiting scalable quality assessment in surgical care. Objective: This study aimed to assess the capability of open-weight and proprietary large language models (LLMs) to classify postoperative complications according to the Clavien-Dindo system using discharge letters, benchmarked against expert annotation. Methods: We analyzed discharge letters from 650 surgical cases of 649 patients (median 67, IQR 58-73 y; 229/649, 35% female) who underwent hepatobiliary surgery between 2010 and 2024. The cohort included grade I-II complications in 24% (153/650), grade III-IV in 19% (121/650), and grade V (death) in 6% (42/650) of patients. A total of 4 open-weight (Qwen3-235B [Alibaba Cloud], Llama-3.3-70B [Meta AI], GPT-OSS-120B [OpenAI], Ministral-3-8B [Mistral AI]) and 2 proprietary (GPT 5.1 [OpenAI], Gemini 3 Pro [Google]) LLMs were prompted to infer complication grades directly from the discharge letters in a zero-shot setting. Model performance was evaluated against expert assessment using accuracy, F1-scores, and Cohen κ. To assess interrater reliability and establish a human benchmark, a stratified 10% (n=65) subset was independently annotated by a second clinician, and Cohen κ was calculated between annotators and between each model and the primary expert. Results: Interrater agreement between the 2 clinical annotators yielded a Cohen κ of 0.75, providing a human benchmark for model performance interpretation. On the full 650-case dataset, open-weight models achieved accuracies ranging from 0.75 to 0.78 for fine-grained prediction, with weighted F1-scores of 0.76-0.78 and macroaveraged F1-scores of 0.50-0.63. For binary classification, accuracies ranged from 0.93 to 0.94, with weighted F1-scores of 0.93-0.95 and Cohen κ of 0.76-0.79, approaching the human interrater benchmark. On a balanced 50-case subset, used as the sole basis for direct cross-model comparison, proprietary models achieved accuracies of 0.78 for fine-grained and 0.94-0.98 for binary classification. An ensemble approach yielded additional gains in classification performance. Conclusions: LLMs demonstrated promising accuracy in classifying postoperative complications from discharge letters in a zero-shot setting, with performance approaching the upper bound of human interrater agreement. Open-weight models offer a particularly attractive trade-off between accuracy and computational efficiency, while ensemble strategies further enhance robustness. These results support the potential of LLMs to standardize complication assessment at scale and enable data-driven quality monitoring in surgical care.

Background: Unstructured clinical text remains a major barrier to interoperable data reuse and large-scale secondary analysis in health care. Large language models (LLMs) have the potential to automate the extraction of structured clinical information; however, their application is limited by the scarcity of high-quality annotated training data. Objective: To address these limitations, this study aims to develop and validate a scalable, privacy-preserving framework that uses synthetic data generated from structured Fast Healthcare Interoperability Resources (FHIR) to fine-tune open-source LLMs for the effective extraction of interoperable clinical information from unstructured text. Methods: We evaluated an LLM-based framework for extracting structured clinical information from cancer-related discharge letters and mapping it to representations compatible with FHIR. To enable large-scale supervised training, we developed a random sample generator that creates synthetic discharge letters using Qwen3-235B by randomly sampling and aggregating structured FHIR data from 41,175 patients with cancer. The resulting synthetic discharge letters (n=75,000) were paired with their originating structured data, forming a large-scale dataset for fine-tuning MedGemma 27B, a 27-billion-parameter medical language model. Evaluation was conducted on the synthetic test dataset (n=7500), real-world discharge letters (n=30), which were evaluated by physicians and a medical student, and a comparative one-shot approach using open-source models (Qwen3, LLaMA, and GPT-OSS). Results: The fine-tuned model achieved high extraction performance across multiple clinical entities on the synthetic test set, with F ₁ -scores of 0.84 for full International Classification of Diseases diagnosis codes, 0.99 for tumor-related information, 0.99 for laboratory values, 0.99 for medication names and dosages, and 0.94 for Anatomical Therapeutic Chemical medication codes. The extraction of procedure-related information was more challenging, with F ₁ -scores of 0.63 for OPS codes and 0.90 for procedure descriptions. The fine-tuned model consistently outperformed general-purpose LLMs in a one-shot comparison across nearly all extraction categories. When evaluated by physicians on real-world discharge letters, the model achieved case-level correctness rates of 78.9% for International Classification of Diseases diagnoses, 86.1% for tumor-related information, 93.0% for medications, and 61.3% for procedures. Conclusions: These results demonstrate that synthetic text generation from structured clinical data enables the effective and scalable training of LLMs for extracting interoperable, multientity clinical information from unstructured documentation.

BACKGROUND Medical speech is a central interface for numerous tasks in modern healthcare workflows, including documentation, report generation, and structured clinical communication. Although documentation is a cornerstone of clinical care, it imposes a substantial administrative burden on health care providers. Automatic Speech Recognition (ASR) offers a promising solution for real-time documentation. However, general-purpose models such as OpenAI’s Whisper struggle with the linguistic density, domain-specific terminology, and structural conventions characteristic of clinical dictation. Moreover, the development of high-performance medical ASR is constrained by a persistent privacy–utility bottleneck, as access to large-scale, annotated clinical audio data is limited by stringent data protection regulations. OBJECTIVE This study aims to develop and evaluate a privacy-preserving framework for fine-tuning ASR models on synthetic clinical speech data, eliminating the need for sensitive patient recordings. We investigate whether LLM-generated, terminology-grounded audio can bridge the performance gap between general-purpose ASR and the demands of real-world clinical dictation across medical specialties. METHODS We present MumbleMED, a privacy-by-design framework for scalable and configurable fine-tuning of ASR models. Realistic clinical narratives were synthesized using Large Language Models (LLMs) based on standardized coding systems (ICD-10, OPS, and RadLex) and converted into high-quality audio via a German Text-to-Speech (TTS) system cloning institutional voices. The resulting pipeline was used to fine-tune Whisper model variants (Tiny to Large v2) on 33,898 synthetic samples (32 hours of audio). Performance was evaluated against a gold-standard benchmark comprising 175 professionally recorded clinical reports from five medical specialties, including radiology, pathology, discharge letters, progress notes and surgical reports. RESULTS Fine-tuning substantially reduced transcription error rates across all model sizes. On the Combined TTS test dataset, the best-performing model, Combined MumbleMED based on Whisper Large v2, reduced WER from 36.71% (baseline Whisper Large v2) to 8.20% and CER from 21.07% to 4.20%. The baseline Whisper Large v2 model achieved a Word Error Rate (WER) of 68.22% on authentic clinical dictation, which decreased to 28.90% after MumbleMED fine-tuning, representing an absolute reduction of nearly 40%. Stratified analyses revealed the lowest error rates in surgical and radiology reports (WER ~19%), whereas discharge letters remained more challenging. In a focused evaluation of 6,576 medical terms, term-specific WER decreased from 38.45% to 13.31%. Qualitative assessment showed that models learned to correctly map verbalized punctuation to symbols and normalize complex clinical units. CONCLUSIONS MumbleMED demonstrates that synthetic clinical speech can effectively bridge the domain gap in ASR performance while remaining compliant with regulatory requirements. By enabling local deployment of high-accuracy ASR within secure clinical infrastructures, the framework provides a practical and privacy-preserving pathway for adapting transcription models to clinical needs.

Background: Embedding models are critical components of Retrieval Augmented Generation (RAG) systems for retrieving and searching unstructured medical data. However, existing models are predominantly trained on publicly available English datasets, limiting their effectiveness in non-English health care settings. More importantly, these models lack training on real-world clinical documents, leading to inaccurate context retrieval when integrated into RAG systems for health care applications. This gap is particularly pronounced in specialized medical documentation containing domain-specific terminology, abbreviations, and nuanced clinical language. Objective: This retrospective study aimed to develop and validate embedding models specifically trained on real-world clinical documents from multiple medical specialties to improve medical information retrieval (IR) and RAG system performance in both German and English language contexts. Methods: We fine-tuned embedding models, so-called sentence transformers, using the multilingual-e5-large architecture as a foundation. Training data consisted of approximately 11 million question-answer pairs synthetically generated from 400,000 diverse clinical documents from a large German tertiary hospital, spanning 163,840 patients and 282,728 clinical cases between 2018 and 2023. The SauerkrautLM-SOLAR-Instruct large language model generated medically relevant questions and corresponding answers for each document. The dataset was additionally pseudonymized and translated into English to aim for broader applicability. Models were evaluated in 2 distinct scenarios: IR using questions with multiple relevant passages, and RAG system performance in both cross-patient and patient-centered contexts. Results: In the IR evaluation, the fine-tuned miracle model achieved a mAP@100 of 0.27, outperforming the multilingual-e5-large baseline (0.14) and state-of-the-art models such as bge-m3 (0.11). In the RAG evaluation, the model demonstrated robust performance comparable with the baseline in the constrained patient-centered scenario (BERTScore F1 0.781 vs 0.778) and showed moderate improvements in the unconstrained cross-patient setting (BLEURT 0.56 vs 0.53). Notably, the model trained on pseudonymized data achieved comparable retrieval performance (mAP@100 0.25) and the highest scores for patient-centered contextual precision (0.93). Performance gains were robust in the German dataset, while the translated English model demonstrated promising results as a proof of concept for cross-lingual transfer. Conclusions: By leveraging a comprehensive real-world dataset spanning multiple medical specialties and using large language models for synthetic question generation, we successfully created and validated domain-specific embedding models. These models can improve medical IR in large-scale search spaces and perform competitively in constrained RAG applications. By publishing the models trained on pseudonymized data, other health care institutions can integrate or adapt these embedding models to their needs. This work establishes a reproducible framework for developing domain-specific clinical embedding models, with the potential to improve data retrieval in medical settings.

BACKGROUND Large Language models (LLMs) have gained significant importance in the medical field, though their role in nursing remains limited at present. LLMs could support summarizing unstructured patient data and identifying relevant risk factors, with potential practical benefits for nursing care. OBJECTIVE This study aims to systematically evaluate the applicability and usefulness of LLMs for German-language nursing practice in inpatient acute care, using selected use cases. METHODS A mixed-methods study was conducted, guided by the QUEST framework and reported in accordance with TRIPOD-LLM. In the qualitative phase, 4 use cases were defined in collaboration with nursing experts. These include summarization tasks (e.g., shift handovers, surgical reports) and risk prediction tasks (e.g., falls, delirium). The PDSQI-9 was selected as the instrument for human evaluation. A codebook was developed, raters were trained, a Delphi pilot study was performed, and the use cases were evaluated. Human ratings and inter-rater reliability (ICC (3,k), AC1/2, Krippendorff's alpha) were assessed. The model's performance was evaluated using classification metrics in quantitative analyses. The study used nursing notes as the primary data source and analyzed them either within the first 24 hours after admission or longitudinally, depending on the use case. The LLM was prompted using a zero-shot setting to assess the ability to extract or infer clinically relevant risk indicators. RESULTS No significant differences in demographic characteristics were observed across the use cases. Report length varied (7–1061 words; 103–14,345 tokens) but was not significantly associated with prediction accuracy in the risk use cases or with rater outcomes across all use cases. In the risk use cases, identifying both fall risk and delirium risk was challenging for nurses, with better performance for delirium risk. In contrast, the Qwen3 model showed substantially higher sensitivity but markedly lower specificity. For both risk factors, PPV (positive predictive value) was slightly lower than in the nursing assessment use case, while NPV (negative predictive value) was slightly higher. In the delirium use case, Qwen3 flagged delirium risk more often. Despite a codebook and rater training, human evaluation showed limited inter-rater reliability. Nurses assigned higher PDSQI-9 scores to LLM outputs in summarization use cases than in risk prediction use cases. While nursing experts recognized the potential of LLMs, they emphasized the need for fine-tuning, more transparent prompts/explanations, and additional clinical context, given inconsistent and sometimes incorrect outputs and interpretability issues CONCLUSIONS This study shows that LLMs have potential for nursing applications, but this potential varies substantially across use cases. To facilitate meaningful, workflow-aligned implementation, it is imperative to employ a human-in-the-loop approach across all stages of the process, from prompt development to fine-tuning and evaluation. Evaluations that utilize real-world data in the target language have been shown to be particularly informative. CLINICALTRIAL The approval for this retroNursing, Large Language Models, inpatient, zero-shot, validation study, Risk Scoringspective study was obtained by the Ethics Committee of the University Hospital Essen (approval number 23-11557-BO).

BACKGROUND The standardized extraction of postoperative complications from unstructured routine clinical documentation remains a major unresolved challenge in digital surgery and health informatics. Although the Clavien–Dindo classification is the established standard for grading postoperative complications, its application in routine clinical documentation is largely implicit and unstructured. OBJECTIVE To assess the capability of open-weight and proprietary large language models (LLMs) to classify postoperative complications according to the Clavien–Dindo system using discharge letters, benchmarked against expert consensus annotations. METHODS We analyzed discharge letters from 650 surgical cases of patients who underwent hepatobiliary surgery between 2010 and 2024. The cohort included Grade I–II complications in 24%, Grade III–IV in 19%, and Grade V (death) in 6% of patients. Representative open-weight (Qwen 3, Llama 3.3, Ministral 3, GPT-OSS) and proprietary (GPT 5.1, Gemini 3 Pro) LLMs were prompted to infer complication grades directly from the discharge letters. Model performance was evaluated against expert assessment using accuracy and a detailed deviation analysis. RESULTS All models were capable of identifying and classifying complications from the unstructured documentation in the discharge letters. On the full 650-case dataset, open-weight models achieved accuracies up to 0.775 for fine-grading prediction and 0.94 for binary classification. On a balanced 50-case subset, proprietary models achieved the highest performance, with accuracies of 0.78 for fine-grading and 0.98 for binary classification. In contrast, open-weight models reached accuracies up to 0.70 in fine-grading and 0.94 for binary grading, with lower computational requirements. An ensemble approach yielded additional gains in classification performance. CONCLUSIONS LLMs can accurately classify postoperative complications from discharge letters, enabling scalable and objective monitoring of surgical outcomes. Their use may reduce manual abstraction workload and promote consistent, data-driven quality assessment in surgical care.

BACKGROUND Unstructured clinical text remains a major barrier to interoperable data reuse and large-scale secondary analysis in healthcare. Large language models (LLMs) have the potential to automate the extraction of structured clinical information; however, their application is limited by the scarcity of high-quality annotated training data. OBJECTIVE - METHODS We evaluated an LLM–based pipeline for extracting structured clinical information from cancer-related discharge letters and mapping it to representations compatible with Fast Healthcare Interoperability Resources (FHIR). To enable large-scale supervised training, we developed a random sample generator that creates synthetic discharge letters using Qwen3 235B by randomly sampling and aggregating structured FHIR data from 41,175 cancer patients. The resulting synthetic discharge letters (n=75k) were paired with their originating structured data, forming a large-scale dataset for fine-tuning MedGemma 27B. Evaluation was conducted on the synthetic test dataset (n=7,500), real-world discharge letters (n=30) which are evaluated by physicians and a medical student, and a comparative one-shot approach using open-source models (Qwen3, LLaMA, and GPT-OSS). RESULTS The fine-tuned model achieved high extraction performance across multiple clinical entities, including full ICD diagnosis codes (F1 = 0.84), tumor-related information (0.99), laboratory values (0.99), medication names and dosages (0.99), and ATC medication codes (0.94). Extraction of procedure-related information was more challenging but remained reliable, with F1 scores of 0.63 for OPS codes and 0.90 for procedure descriptions. In a one-shot comparison of general-purpose LLMs with the fine-tuned model, the fine-tuned model consistently outperformed general-purpose LLMs in nearly all extraction categories. When applied to real-world discharge letters, performance remained robust, with F1 scores of 78.9% for ICD diagnoses, 86.1% for tumor-related information, 93% for medications, and 61.3% for procedures. CONCLUSIONS These results demonstrate that synthetic text generation from structured clinical data enables effective and scalable training of LLMs for extracting interoperable, multi-entity clinical information from unstructured documentation.