RNA-guided contrastive learning enhances patient-level prediction from histology
Preprint
Abstract
Predicting molecular receptor status, including estrogen receptor (ER), progesterone receptor (PR), and human epidermal growth factor receptor 2 (HER2), directly from routine hematoxylin and eosin (H&E)-stained whole-slide images (WSIs) could reduce reliance on costly immunohistochemistry, but image-only models have no access to the transcriptomic programs that define these subtypes. Here we show that coupling a whole-slide vision transformer (GigaPath) with bulk RNA-sequencing profiles through a training-time-only contrastive objective improves receptor-status classification from WSIs alone, without requiring any RNA-seq data at inference. Rather than training a dedicated bulk-transcriptomics encoder, we purpose the gene-sentence encoding and pretrained text encoder from BioBERT 1 and apply them to bulk RNA-seq profiles from The Cancer Genome Atlas breast cancer cohort (TCGA-BRCA). Under five-fold cross-validation, this RNA-guided pre-training consistently improved AUROC for ER, PR, and HER2 relative to a contrastive-pre-training-free baseline and reproducibly introduced receptor-relevant structure into the learned slide representation. However, evaluation on independent external cohorts revealed a striking, receptor-dependent divergence: gains for ER and PR, both governed by broad, multi-gene luminal transcriptional programs, generalized robustly, whereas the apparent AUROC improvement for HER2, driven by focal ERBB2 amplification rather than a coordinated transcriptional signature, masked a severe loss of classification sensitivity at the operating threshold. These findings establish training-time transcriptomic supervision as a practical route to enhancing histology-only molecular subtyping, while showing that gene-selection strategy, and not model architecture alone, determines which molecular alterations such supervision can teach a vision model to recognize.Keywords
This publication has 18 references indexed in Scilit:
- A visual–omics foundation model to bridge histopathology with spatial transcriptomicsNature Methods, 2025
- BulkRNABert: Cancer prognosis from bulk RNA-seq based language modelsbioRxiv, 2024
- A whole-slide foundation model for digital pathology from real-world dataNature, 2024
- GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPTbioRxiv, 2023
- Cell2Sentence: Teaching Large Language Models the Language of BiologybioRxiv, 2023
- LiT: Zero-Shot Transfer with Locked-image text Tuning2022
- BioBERT: a pre-trained biomedical language representation model for biomedical text miningBioinformatics, 2019
- Comprehensive Molecular Portraits of Invasive Lobular Breast CancerCell, 2015
- Comprehensive molecular portraits of human breast tumoursNature, 2012
- Integrated genomic analyses of ovarian carcinomaNature, 2011