Dual-Stream Transformer with Cross-Attention on Whole-Slide Image Pyramids for Cancer Prognosis
Authors:
- Pei LiuPLPei Liu
- Bo FuBFBo Fu
- Feng YeFYFeng Ye
- Rui YangRYRui Yang
- Bin XuBXBin Xu
- Luping JiLJLuping Ji
Preprint
Abstract
The cancer prognosis on gigapixel Whole-Slide Images (WSIs) has always been a challenging task. Most existing approaches focus solely on single-resolution images. The multi-resolution schemes, utilizing image pyramids to enhance WSI visual representations, have not yet been paid enough attention to. In order to explore a multi-resolution solution for improving cancer prognosis accuracy, this paper proposes a dual-stream architecture to model WSIs by an image pyramid strategy. This architecture consists of two sub-streams: one for low-resolution WSIs, and the other especially for high-resolution ones. Compared to other approaches, our scheme has three highlights: (i) there exists a one-to-one relation between stream and resolution; (ii) a square pooling layer is added to align the patches from two resolution streams, largely reducing computation cost and enabling a natural stream feature fusion; (iii) a cross-attention-based method is proposed to pool high-resolution patches spatially under the guidance of low-resolution ones. We validate our scheme on three publicly-available datasets with a total number of 3,101 WSIs from 1,911 patients. Experimental results verify that (i) hierarchical dual-stream representation is more effective than single-stream ones for cancer prognosis, gaining an average C-Index rise of 5.0% and 1.8% on a single low-resolution and high-resolution stream, respectively; (ii) our dual-stream scheme could outperform current state-of-the-art ones, by an average C-Index improvement of 5.1%; (iii) the cancer diseases with observable survival differences could have different preferences for model complexity. Our scheme could serve as an alternative tool for further facilitating WSI prognosis research.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Hierarchical graph representations in digital pathologyMedical Image Analysis, 2022
- Data-efficient and weakly supervised computational pathology on whole-slide imagesNature Biomedical Engineering, 2021
- Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networksMedical Image Analysis, 2020
- Deep learning for prediction of colorectal cancer outcome: a discovery and validation studyThe Lancet, 2020
- Artificial intelligence in digital pathology — new tools for diagnosis and precision oncologyNature Reviews Clinical Oncology, 2019
- Multiple instance learning: A survey of problem characteristics and applicationsPattern Recognition, 2018
- Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresNature Communications, 2016
- A Contemporary Prostate Cancer Grading System: A Validated Alternative to the Gleason ScoreEuropean Urology, 2016
- Toward understanding and exploiting tumor heterogeneityNature Medicine, 2015
- Biomarkers in Cancer Staging, Prognosis and Treatment SelectionNature Reviews Cancer, 2005