Katarzyna Borys

Large language models are increasingly being used to evaluate clinical texts, a concept known as LLM-as-a-judge. However, most of the existing evidence focuses on English data and uses pooled agreement metrics that obscure clinically relevant failure modes. This study audited six open-weight models against expert judges on four German clinical datasets covering report and nursing documentation summarization, discharge summary generation, and patient letter generation. The models were evaluated across three paradigms and four prompt configurations. Alignment with experts varied by criterion and corpus, and no configuration proved reliable across paradigms. All judges exhibited position and identity bias. Substituting LLM-as-a-judge for the expert panel shifted 57.2% of pointwise categories, swapped 41.7% of pairwise winners, missed 10.2% of clinical errors, and inverted clinical priorities by overrating the criteria succinct and accurate. Therefore, LLM-as-a-judge is not a safe substitute for experts in German clinical text evaluations, and deployment thresholds should follow decision-impact analysis for LLM-as-a-judge in clinical contexts.

Background: Continuous patient monitoring generates vast volumes of high-frequency monitoring data, such as electrocardiogram, electroencephalography, and invasive blood pressure tracings, that are often persisted in heterogeneous, noninteroperable formats, limiting integration with electronic health records and secondary use. Although Fast Healthcare Interoperability Resources (FHIR) supports general health data exchange, native support for high-frequency medical monitoring data, including associated metadata and data quality metrics, remains limited. Objective: This study aims to develop a modality-agnostic FHIR package implementing a Metadata-Proxy pattern to enable scalable, semantically precise representation of high-frequency medical monitoring data, decoupling structured clinical metadata from raw signal storage. Methods: This study provides the FHIR package Time Series Integration and Data in Endpoints (TIDE), a collection of FHIR Release 4 artifacts for interoperable exchange and the standardization of high-frequency monitoring clinical data, using electroencephalography as the primary use case while remaining applicable to generic medical monitoring data. The core TIDE Observation profile separates technical metadata and analytical summaries stored directly in FHIR from high-volume raw data referenced via Endpoint resources. Quantitative results are standardized with pattern slicing, including sampling frequency, statistical summaries, and data quality metrics. Semantic interoperability was achieved through a hybrid terminology binding, using Systematized Nomenclature of Medicine Clinical Terms, Logical Observation Identifiers Names and Codes for modalities, and custom CodeSystems for technical metadata, waveform modalities, channel identifiers, and connection types not covered by standard terminologies. All profiles were authored in FHIR Shorthand and compiled with SUSHI to produce a validated Implementation Guide conformant with the FHIR Release 4 core specification. Results: The resulting framework provides a modality-agnostic structure for high-frequency monitoring data, comprising 7 primary FHIR artifacts that define 16 distinct component slices for metadata and signal analysis results. Observation resources reference associated Device and Endpoint instances, encapsulate acquisition parameters, and link to full-time-series repositories. Controlled vocabularies ensure unambiguous exchange, thereby enabling downstream use in research and decision support. The architecture decouples the metadata layer from the storage layer via a standardized Endpoint extension, enabling efficient querying of signals without retrieving large amounts of raw high-frequency monitoring data. Conclusions: TIDE provides a reusable, standards-compliant framework for integrating continuous high-frequency monitoring data into FHIR-based ecosystems. By separating semantically rich, queryable metadata from raw signal storage, TIDE enables longitudinal patient monitoring and automated signal quality-based triage without requiring modification of existing signal storage infrastructure.

This study evaluated a user-centered clinical dashboard using FHIR-Server data to improve information accessibility for physicians treating melanoma patients. Current electronic health record systems distribute critical information across multiple modules and prioritize billing and research over clinical decision-making. We employed a mixed-methods approach combining think-aloud protocols, qualitative content analysis, and standardized usability questionnaires across three iterative evaluation cycles with nine participants in eleven sessions. Our dashboard implementation demonstrated substantial workflow improvements, reducing required interactions from 17 to 23 clicks to just 5–6 clicks, enabling physicians to focus more on treatment decisions than information gathering. The Post-Study System Usability Questionnaire showed progressive enhancement across iterations, with overall scores improving from 2.4 to 1.8. Affinity diagrams revealed key improvements in information accessibility and workflow integration. These findings indicate the value of a user-centered, holistic approach to clinical dashboard development, showing that FHIR-based interoperability infrastructures can be transformed into point-of-care applications when information is structured around clinicians’ mental models rather than technically derived data structures. Beyond the melanoma context, this approach provides a transferable foundation for systematically exploring clinical contexts and translating these insights into meaningful, workflow-integrated applications across clinical domains.

Background: Although Digital Imaging and Communications in Medicine (DICOM) metadata are widely used to manage medical imaging data and support clinical workflows, their suitability as a sole basis for automatic computed tomography (CT) series labeling and characterization is limited. DICOM metadata are frequently inconsistently populated, institution specific, use unregulated private tags, and have variable reliability even within standardized fields. Consequently, automated series selection for downstream AI applications often remains unreliable, necessitating manual curation within clinical workflows. Objective: This study presents Orchestrate, a modular AI framework for automated orchestration of CT imaging data. By integrating a hierarchy of deep learning models, Orchestrate enables pixel-level classification and routing of CT series and accurate metadata-independent identification of anatomical regions, contrast-enhanced series, and reconstruction kernels, supporting seamless downstream AI integration without manual curation. Methods: Orchestrate combined 3 pretrained models for anatomical regions, landmarks, and body part classification and 4 newly developed You Only Look Once (YOLO) v8–based models to classify contrast enhancement, recognize reconstruction kernels, and infer laterality. Three datasets were used: an internal development dataset comprising 27,418 CT studies for individual model development, an internal framework evaluation dataset comprising 200 CT studies to assess the complete framework under a simulated real-world scenario, and an external dataset from The Cancer Imaging Archive comprising 100 CT studies. For the internal development data, reference standards were derived from complete and unambiguous DICOM metadata following institutional definitions. For the 2 evaluation datasets, 3 radiographers independently established reference standards as DICOM metadata were not assumed to be complete or consistent. Clinical utility was assessed through cohort selection tasks involving 3 predefined target cases, with 3 radiologists reviewing selection accuracy. The interrater agreement was assessed using the Fleiss κ. The model performance was evaluated using F1-scores. Results: DICOM metadata were incomplete in the internal framework evaluation dataset and external dataset, with missing rates of 42.9% (413/963) and 92.9% (105/113) for contrast enhancement, respectively, and a missing reconstruction kernel information rate of 0.4% (4/963) in the internal framework evaluation dataset. During model development, individual models achieved macro–F1-scores ranging from 0.982 to 0.989. At the framework level, Orchestrate achieved high classification performance across internal (weighted F1-score ranged from 0.920 to 1.000; macro–F1-score ranged from 0.879 to 1.000) and external (weighted F1-score ranged from 0.946 to 1.000; macro–F1-score ranged from 0.777 to 0.929) cohorts. For clinical use cases, the overall selection accuracy was 97.7% (217/222). Conclusions: Orchestrate enables automated pixel-based classification, detection, and semantic description of CT series, reducing reliance on manual selection and the risk of inconsistent metadata. By generating standardized semantic content, the framework provides a proof of concept for improving interoperability with clinical systems and supports the reliable, reproducible integration of AI-driven imaging pipelines into clinical workflows.

Objectives: To evaluate whether CycleGAN-based synthesis of T2-HASTE images from routinely acquired fast whole-body localizer (FWBL) can support accelerated body composition analysis (BCA) by preserving sufficient information for an established T2-HASTE–based BCA pipeline to generate measurements comparable to those derived from reference T2-HASTE images. Materials and Methods: This retrospective single-center proof-of-concept study included 468 patients (53% female, mean age 51±18 y) who underwent WB-MRI with FWBL and T2-HASTE between March 2012 and December 2025, 431 (92%) of whom were examined from 2023 onward. A CycleGAN model was trained to generate synthetic T2-HASTE images from registered FWBL and T2-HASTE images. Fully automated MRI-based BCA was conducted on synthetic and reference T2-HASTE images using a fixed pretrained BCA model, focusing on 5 key segmentation compartments: subcutaneous adipose tissue (SAT), muscle, abdominal cavity, thoracic cavity, and bone, with subsequent region-specific volumetric analyses of SAT, muscle, and bone in the thoracic, abdominal, and whole-body regions. Derived BCA markers included the Sarcopenia Index (muscle-to-bone ratio) and the muscle-to-SAT ratio. Agreement was assessed using Bland-Altman analysis, including mean bias expressed in absolute units and as a percentage of the mean reference volume, median absolute percentage error, Pearson correlation, and weighted Cohen kappa for tertile-based stratification. Segmentation overlap was evaluated using Sørensen-Dice similarity coefficients. In addition, a qualitative reader evaluation was performed by 4 radiologists across 30 cases. Results: Mean percentage bias across the thoracic, abdominal, and whole-body regions was −0.05%, 3.51%, and 2.91% for muscle, −0.19%, −1.21%, and 0.71% for SAT, and −7.10%, −8.86%, and −6.30% for bone, respectively. Median absolute percentage error remained low for muscle (2.96% to 3.52%) and SAT (2.03% to 3.71%), whereas bone showed the highest relative error (5.93% to 8.26%). Tertile-based stratification agreement was high, particularly for the muscle-to-SAT ratio, with unchanged tertile assignment in 89.4% of thoracic, 93.6% of abdominal, and 95.7% of whole-body analyses (κ=0.92, 0.95, and 0.97, respectively). For the Sarcopenia Index, unchanged tertile assignment was 68.1%, 80.9%, and 87.2%, respectively (κ=0.76, 0.86, and 0.90). Segmentation overlap was high across the main compartments, with mean Sørensen-Dice coefficients of 0.90 for SAT, 0.87 for muscle, 0.94 for the abdominal cavity, and 0.91 for the thoracic cavity, whereas bone showed lower overlap (0.76). Qualitative evaluation showed positive ratings for both synthetic image utility and segmentation plausibility, with the synthetic T2 representation rated as providing added value over FWBL for coarse body composition assessment on a 5-point Likert scale, where 1 indicated strong disagreement, and 5 indicated strong agreement (mean: 4.52, median: 5). Conclusions: Combining FWBL MRI with CycleGAN-based image synthesis yielded BCA measurements that closely matched those from reference T2-HASTE images and largely preserved body composition stratification. These findings demonstrate the feasibility of deriving quantitative BCA information from routinely acquired WB-MRI localizer data and support further investigation of accelerated MRI-based BCA approaches using simplified acquisitions.

Background: The standardized extraction of postoperative complications from unstructured routine clinical documentation remains a major unresolved challenge in digital surgery and health informatics. Although the Clavien-Dindo classification is the established standard for grading postoperative complications, its application in routine clinical documentation is largely implicit and unstructured, limiting scalable quality assessment in surgical care. Objective: This study aimed to assess the capability of open-weight and proprietary large language models (LLMs) to classify postoperative complications according to the Clavien-Dindo system using discharge letters, benchmarked against expert annotation. Methods: We analyzed discharge letters from 650 surgical cases of 649 patients (median 67, IQR 58-73 y; 229/649, 35% female) who underwent hepatobiliary surgery between 2010 and 2024. The cohort included grade I-II complications in 24% (153/650), grade III-IV in 19% (121/650), and grade V (death) in 6% (42/650) of patients. A total of 4 open-weight (Qwen3-235B [Alibaba Cloud], Llama-3.3-70B [Meta AI], GPT-OSS-120B [OpenAI], Ministral-3-8B [Mistral AI]) and 2 proprietary (GPT 5.1 [OpenAI], Gemini 3 Pro [Google]) LLMs were prompted to infer complication grades directly from the discharge letters in a zero-shot setting. Model performance was evaluated against expert assessment using accuracy, F1-scores, and Cohen κ. To assess interrater reliability and establish a human benchmark, a stratified 10% (n=65) subset was independently annotated by a second clinician, and Cohen κ was calculated between annotators and between each model and the primary expert. Results: Interrater agreement between the 2 clinical annotators yielded a Cohen κ of 0.75, providing a human benchmark for model performance interpretation. On the full 650-case dataset, open-weight models achieved accuracies ranging from 0.75 to 0.78 for fine-grained prediction, with weighted F1-scores of 0.76-0.78 and macroaveraged F1-scores of 0.50-0.63. For binary classification, accuracies ranged from 0.93 to 0.94, with weighted F1-scores of 0.93-0.95 and Cohen κ of 0.76-0.79, approaching the human interrater benchmark. On a balanced 50-case subset, used as the sole basis for direct cross-model comparison, proprietary models achieved accuracies of 0.78 for fine-grained and 0.94-0.98 for binary classification. An ensemble approach yielded additional gains in classification performance. Conclusions: LLMs demonstrated promising accuracy in classifying postoperative complications from discharge letters in a zero-shot setting, with performance approaching the upper bound of human interrater agreement. Open-weight models offer a particularly attractive trade-off between accuracy and computational efficiency, while ensemble strategies further enhance robustness. These results support the potential of LLMs to standardize complication assessment at scale and enable data-driven quality monitoring in surgical care.

Background: Oesophageal cancer remains the sixth most lethal malignancy, with 5‐year survival rates around 22%. CT‐derived 2D body composition analysis has emerged as a promising prognostic tool, but conventional, often manually created single‐slice L3 measurements are unsuited for future clinical implementation. We evaluated fully automated AI‐derived volumetric body composition indices as prognostic parameters using conventional L3 and BMI measurements as reference. Methods: This retrospective cohort study included patients with histologically confirmed oesophageal cancer treated between 2011 and 2024. Automated deep learning segmentation using the nnU‐Net‐based body and organ analysis pipeline quantified abdominal tissue volumes from staging CTs. Three normalized indices were calculated: sarcopenia index (SI, muscle/bone volume ratio), myosteatotic fat index (MFI, intramuscular/total adipose tissue volume ratio) and abdominal fat index (AFI, visceral/subcutaneous adipose tissue volume ratio). Cox proportional hazards models assessed prognostic value after sequential adjustment for age, sex, metastatic status, ECOG performance status, BMI and L3‐derived indices. Kaplan–Meier analysis evaluated survival differences stratified by sex‐specific median values. Results: The cohort comprised 563 patients (19.7% female), median age 65.4 years (IQR: 58.8–71.3); 10.1% ( n = 57) had metastatic disease and 68.7% ( n = 387) underwent surgery. Males had higher sarcopenia index (2.54 ± 0.41 vs. 2.24 ± 0.45, p < 0.001) and abdominal fat index (median 0.72 vs. 0.36, p < 0.001), whereas MFI showed no difference ( p = 0.89). During median follow‐up of 22 months, 367 deaths (65.9%) occurred. Median overall survival was 28.6 months (95% CI: 23.7–33.3); 1‐year survival 70.7% (95% CI: 67.2%–74.8%) and 5‐year survival 32.8% (95% CI: 29.6%–38.3%), with marked differences by metastatic status (M0 vs. M1 1‐year survival: 73.4% vs. 46.4%). In the fully adjusted model incorporating all three volumetric indices alongside clinical covariates, BMI and L3‐derived parameters ( n = 532), only sarcopenia index retained independent significance (HR = 0.56, 95% CI: 0.37–0.82, p = 0.003); all others showed p ≥ 0.26. Metastatic status (HR = 2.23, p < 0.001) and ECOG (HR = 1.29, p < 0.001) remained significant. All L3 indices lost significance alongside volumetric parameters ( p > 0.14). Male patients with high sarcopenia index showed longer survival compared with patients with low sarcopenia index (33.3 vs. 21.8 months, p < 0.001); female patients showed larger descriptive contrasts (68.8 vs. 16.8 months, p < 0.001) that were formally confirmed by a significant sex‐SI Cox interaction (LRT p = 0.026). Conclusions: Automatic AI‐derived volumetric body composition parameters calculated from routine staging CTs predict overall survival in patients with oesophageal cancer and are associated with sex‐related differences of prognostic impact, supporting further evaluation toward future clinical use.

Background: Unstructured clinical text remains a major barrier to interoperable data reuse and large-scale secondary analysis in health care. Large language models (LLMs) have the potential to automate the extraction of structured clinical information; however, their application is limited by the scarcity of high-quality annotated training data. Objective: To address these limitations, this study aims to develop and validate a scalable, privacy-preserving framework that uses synthetic data generated from structured Fast Healthcare Interoperability Resources (FHIR) to fine-tune open-source LLMs for the effective extraction of interoperable clinical information from unstructured text. Methods: We evaluated an LLM-based framework for extracting structured clinical information from cancer-related discharge letters and mapping it to representations compatible with FHIR. To enable large-scale supervised training, we developed a random sample generator that creates synthetic discharge letters using Qwen3-235B by randomly sampling and aggregating structured FHIR data from 41,175 patients with cancer. The resulting synthetic discharge letters (n=75,000) were paired with their originating structured data, forming a large-scale dataset for fine-tuning MedGemma 27B, a 27-billion-parameter medical language model. Evaluation was conducted on the synthetic test dataset (n=7500), real-world discharge letters (n=30), which were evaluated by physicians and a medical student, and a comparative one-shot approach using open-source models (Qwen3, LLaMA, and GPT-OSS). Results: The fine-tuned model achieved high extraction performance across multiple clinical entities on the synthetic test set, with F ₁ -scores of 0.84 for full International Classification of Diseases diagnosis codes, 0.99 for tumor-related information, 0.99 for laboratory values, 0.99 for medication names and dosages, and 0.94 for Anatomical Therapeutic Chemical medication codes. The extraction of procedure-related information was more challenging, with F ₁ -scores of 0.63 for OPS codes and 0.90 for procedure descriptions. The fine-tuned model consistently outperformed general-purpose LLMs in a one-shot comparison across nearly all extraction categories. When evaluated by physicians on real-world discharge letters, the model achieved case-level correctness rates of 78.9% for International Classification of Diseases diagnoses, 86.1% for tumor-related information, 93.0% for medications, and 61.3% for procedures. Conclusions: These results demonstrate that synthetic text generation from structured clinical data enables the effective and scalable training of LLMs for extracting interoperable, multientity clinical information from unstructured documentation.

BACKGROUND Medical speech is a central interface for numerous tasks in modern healthcare workflows, including documentation, report generation, and structured clinical communication. Although documentation is a cornerstone of clinical care, it imposes a substantial administrative burden on health care providers. Automatic Speech Recognition (ASR) offers a promising solution for real-time documentation. However, general-purpose models such as OpenAI’s Whisper struggle with the linguistic density, domain-specific terminology, and structural conventions characteristic of clinical dictation. Moreover, the development of high-performance medical ASR is constrained by a persistent privacy–utility bottleneck, as access to large-scale, annotated clinical audio data is limited by stringent data protection regulations. OBJECTIVE This study aims to develop and evaluate a privacy-preserving framework for fine-tuning ASR models on synthetic clinical speech data, eliminating the need for sensitive patient recordings. We investigate whether LLM-generated, terminology-grounded audio can bridge the performance gap between general-purpose ASR and the demands of real-world clinical dictation across medical specialties. METHODS We present MumbleMED, a privacy-by-design framework for scalable and configurable fine-tuning of ASR models. Realistic clinical narratives were synthesized using Large Language Models (LLMs) based on standardized coding systems (ICD-10, OPS, and RadLex) and converted into high-quality audio via a German Text-to-Speech (TTS) system cloning institutional voices. The resulting pipeline was used to fine-tune Whisper model variants (Tiny to Large v2) on 33,898 synthetic samples (32 hours of audio). Performance was evaluated against a gold-standard benchmark comprising 175 professionally recorded clinical reports from five medical specialties, including radiology, pathology, discharge letters, progress notes and surgical reports. RESULTS Fine-tuning substantially reduced transcription error rates across all model sizes. On the Combined TTS test dataset, the best-performing model, Combined MumbleMED based on Whisper Large v2, reduced WER from 36.71% (baseline Whisper Large v2) to 8.20% and CER from 21.07% to 4.20%. The baseline Whisper Large v2 model achieved a Word Error Rate (WER) of 68.22% on authentic clinical dictation, which decreased to 28.90% after MumbleMED fine-tuning, representing an absolute reduction of nearly 40%. Stratified analyses revealed the lowest error rates in surgical and radiology reports (WER ~19%), whereas discharge letters remained more challenging. In a focused evaluation of 6,576 medical terms, term-specific WER decreased from 38.45% to 13.31%. Qualitative assessment showed that models learned to correctly map verbalized punctuation to symbols and normalize complex clinical units. CONCLUSIONS MumbleMED demonstrates that synthetic clinical speech can effectively bridge the domain gap in ASR performance while remaining compliant with regulatory requirements. By enabling local deployment of high-accuracy ASR within secure clinical infrastructures, the framework provides a practical and privacy-preserving pathway for adapting transcription models to clinical needs.