End-to-end multimodal pathology foundation model with clinical dialogue
Top Cited Papers
Open Access
- 31 July 2026
- journal article
- research article
- Published by Springer Nature in Nature Medicine
- Vol. 32 (9) , 3203-3213
Abstract
Recent rapid progress in the field of computational pathology has been enabled by foundation models. These models are beginning to move beyond encoding image patches toward whole-slide understanding, but their clinical utility remains limited. Here we present PRISM2, a multimodal slide-level foundation model trained on 2.3 million whole-slide images and 14 million question−answer pairs derived from 700,000 pathology reports. Through clinical dialogue supervision, PRISM2 aligns histomorphology with diagnostic reasoning, yielding representations that support both prompt-based inference and transferable embeddings for downstream tasks. With prompt-based inference, PRISM2 achieves or exceeds ( P < 0.05) the balanced accuracy of clinical-grade products calibrated for cancer detection in the prostate, breast and breast lymph node. Additionally, across comprehensive diagnostic, biomarker and survival benchmarks, PRISM2 embeddings never statistically underperform previous foundation models via linear probing ( P < 0.05). Furthermore, task-specific fine-tuning on survival prediction outperforms training from scratch on the same large survival dataset. PRISM2 demonstrates how language-supervised pretraining provides a scalable, clinically grounded signal for generalizable pathology representations, bridging human diagnostic reasoning and foundation model performance.Keywords
This publication has 41 references indexed in Scilit:
- Towards robust foundation models for digital pathologyNature Communications, 2026
- A multimodal knowledge-enhanced whole-slide pathology foundation modelNature Communications, 2025
- Multistain Pretraining for Slide Representation Learning in Pathology2024
- Weakly supervised joint whole-slide segmentation and classification in prostate cancerMedical Image Analysis, 2023
- Transformer-based unsupervised contrastive learning for histopathological image classificationMedical Image Analysis, 2022
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question AnsweringInternational Journal of Computer Vision, 2018
- An Integrated TCGA Pan-Cancer Clinical Data Resource to Drive High-Quality Survival Outcome AnalyticsCell, 2018
- Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast CancerJAMA, 2017
- Memorial Sloan Kettering-Integrated Mutation Profiling of Actionable Cancer Targets (MSK-IMPACT)The Journal of Molecular Diagnostics, 2015
- The Cancer Genome Atlas Pan-Cancer analysis projectNature Genetics, 2013