Mireia Crispin-Ortuzar

Large Vision Language Models (LVLMs) are increasingly used in computational pathology for image classification, description generation, question answering and interactive diagnostics. However, most pathology LVLMs analyse small regions of interest rather than pyramidal, gigapixel-scale whole-slide images (WSIs), limiting their use for tasks requiring whole-slide assessment across sub-regions and magnification levels. Here, we present ALPaCA (Adapting Llama for Pathology Context Analysis), a slide-level LVLM framework for WSI question answering across diverse cancer types and tissue sites. ALPaCA is trained using 35,913 WSIs with curated descriptions and 341,051 question-answer pairs from TCGA and GTEx. It combines a LongFormer vision-text adaptor with a Gaussian mixture model-based prototyping adaptor and Llama3.1. ALPaCA exceeds 90% accuracy on internal close-ended benchmarks and maintains 77–82% accuracy on independent external cohorts. Expert pathologist evaluation of open-ended responses supports its slide-level reasoning capability. Additionally, ALPaCA can be fine-tuned on organ- or disease-specific datasets, supporting specialised pathology question answering.

Liver cancer primarily develops in patients with chronic liver disease (CLD), yet most cases are diagnosed at advanced stages with poor prognosis. While CLD surveillance generates extensive longitudinal data, its free-text nature hinders large-scale research. To address this, we developed a scalable framework using open-source LLMs with constrained decoding to process unstructured text across radiology, pathology, and transplant assessment domains. A calibration set comprising 507 reports from 30 patients was manually annotated to benchmark four LLMs against a regular expression baseline across 70 tasks. Llama-3.3-70B performed best, exceeding 90% accuracy on 59/70 tasks, outperforming Llama-3.1-8B (a smaller variant), OpenBioLLM-70B (a medically fine-tuned model), and DeepSeek-R1-8B. Constrained decoding achieved >99.9% format adherence, far surpassing unconstrained prompting (87.4%). Applied to the full cohort, the pipeline analysed 22,493 reports to generate a patient-level database of 29,225 datapoints (35 variables, 835 patients) without manual annotation. Further analysis confirmed known liver cancer risk factors (male sex, viral hepatitis, smoking, diabetes), and allowed for reconstruction of individualised disease timelines. This work provides a scalable blueprint for transforming real-world clinical free-text into structured formats and personalised patient trajectories. Future applications have the potential to accelerate data-driven research into early cancer detection within complex pre-cancerous diseases like CLD.

Spatial quantification is a critical step in most computational pathology tasks, from guiding pathologists to areas of clinical interest to discovering tissue phenotypes behind novel biomarkers. To circumvent the need for manual annotations, modern computational pathology methods have favored multiple-instance learning approaches that can accurately predict whole-slide image labels, albeit at the expense of losing their spatial awareness. Here we prove mathematically that a model using instance-level aggregation could achieve superior spatial quantification without compromising on whole-slide image prediction performance. We then introduce a superpatch-based measurable multiple-instance learning method, SMMILe, and evaluate it across 6 cancer types, 3 highly diverse classification tasks and 8 datasets involving 3,850 whole-slide images. We benchmark SMMILe against nine existing methods using two different encoders—an ImageNet pretrained and a pathology-specific foundation model—and show that in all cases SMMILe matches or exceeds state-of-the-art whole-slide image classification performance while simultaneously achieving outstanding spatial quantification.

Motivation: Survival analysis is a branch of statistics that is crucial in medicine for modeling the time to critical events such as death or relapse, in order to improve treatment strategies and patient outcomes. Selecting survival models often involves a trade-off between performance and interpretability; deep learning models offer high performance but lack the transparency of more traditional approaches. This poses a significant issue in medicine, where practitioners are reluctant to use black-box models for critical patient decisions. Results: We introduce CoxKAN, a Cox proportional hazards Kolmogorov-Arnold Network for interpretable, high-performance survival analysis. Kolmogorov-Arnold Networks (KANs) were recently proposed as an interpretable and accurate alternative to multi-layer perceptrons. We evaluated CoxKAN on four synthetic and nine real datasets, including five cohorts with clinical data and four with genomics biomarkers. In synthetic experiments, CoxKAN accurately recovered interpretable hazard function formulae and excelled in automatic feature selection. Evaluations on real datasets showed that CoxKAN consistently outperformed the traditional Cox proportional hazards model (by up to 4% in C-index) and matched or surpassed the performance of deep learning-based models. Importantly, CoxKAN revealed complex interactions between predictor variables and uncovered symbolic formulae, which are key capabilities that other survival analysis methods lack, to provide clear insights into the impact of key biomarkers on patient risk. Availability and implementation: CoxKAN is available at GitHub and Zenodo.

Venous tumour thrombus (VTT), where the primary tumour invades the renal vein and inferior vena cava, affects 10–15% of renal cell carcinoma (RCC) patients. Curative surgery for VTT is high-risk, but neoadjuvant therapy may improve outcomes. The NAXIVA trial demonstrated a 35% VTT response rate after 8 weeks of neoadjuvant axitinib, a VEGFR-directed therapy. However, understanding non-response is critical for better treatment. Here we show that response to axitinib in this setting is characterised by a distinct and predictable set of features. We conduct a multiparametric investigation of samples collected during NAXIVA using digital pathology, flow cytometry, plasma cytokine profiling and RNA sequencing. Responders have higher baseline microvessel density and increased induction of VEGF-A and PlGF during treatment. A multi-modal machine learning model integrating features predict response with an AUC of 0.868, improving to 0.945 when using features from week 3. Key predictive features include plasma CCL17 and IL-12. These findings may guide future treatment strategies for VTT, improving the clinical management of this challenging scenario.

Large Vision Language Models (LVLMs) have recently revolutionized computational pathology. LVLMs transform pathology image embeddings into tokens recognizable by large language models, facilitating zero-shot image classification, description generation, question answering, and interactive diagnostics. In clinical practice, pathological assessments often require the analysis of entire tissue slides, integrating information from multiple sub-regions and magnification levels. However, existing LVLM frameworks have been restricted to the analysis of small, predefined regions of interest, lacking the ability to analyze pyramidal, gigapixel-scale whole-slide images (WSIs). In this work, we introduce ALPaCA ( A dapting L lama for Pa thology C ontext A nalysis), and train the first general-purpose slide-level LVLM, leveraging 35,913 WSIs with curated descriptions alongside 341,051 question and answer pairs encompassing diverse diagnoses, procedures, and tissue types. By developing LongFormer, a vision-text interactive slide-level adaptor, and integrating it with a Gaussian mixture model-based prototyping adaptor, followed by training with Llama3.1, ALPaCA achieves superior performance in slide-level question answering, achieving over 90% accuracy in close-ended tests and high accuracy in open-ended questions as evaluated by expert pathologists, highlighting its potential for slide-level computer-aided diagnosis systems. Additionally, we show that ALPaCA can be readily fine-tuned on in-depth, organ-specific, or disease-specific datasets, underscoring its adaptability and utility for specialized pathology tasks.