DSCA: A Dual-Stream Network with Cross-Attention on Whole-Slide Image Pyramids for Cancer Prognosis
Authors:
- Pei LiuPLPei Liu
- Bo FuBFBo Fu
- Feng YeFYFeng Ye
- Rui YangRYRui Yang
- Bin XuBXBin Xu
- Luping JiLJLuping Ji
Preprint
Abstract
The cancer prognosis on gigapixel Whole-Slide Images (WSIs) has always been a challenging task. To further enhance WSI visual representations, existing methods have explored image pyramids, instead of single-resolution images, in WSIs. In spite of this, they still face two major problems: high computational cost and the unnoticed semantical gap in multi-resolution feature fusion. To tackle these problems, this paper proposes to efficiently exploit WSI pyramids from a new perspective, the dual-stream network with cross-attention (DSCA). Our key idea is to utilize two sub-streams to process the WSI patches with two resolutions, where a square pooling is devised in a high-resolution stream to significantly reduce computational costs, and a cross-attention-based method is proposed to properly handle the fusion of dual-stream features. We validate our DSCA on three publicly-available datasets with a total number of 3,101 WSIs from 1,911 patients. Our experiments and ablation studies verify that (i) the proposed DSCA could outperform existing state-of-the-art methods in cancer prognosis, by an average C-Index improvement of around 4.6%; (ii) our DSCA network is more efficient in computation -- it has more learnable parameters (6.31M vs. 860.18K) but less computational costs (2.51G vs. 4.94G), compared to a typical existing multi-resolution network. (iii) the key components of DSCA, dual-stream and cross-attention, indeed contribute to our model's performance, gaining an average C-Index rise of around 2.0% while maintaining a relatively-small computational load. Our DSCA could serve as an alternative and effective tool for WSI-based cancer prognosis.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Fractal graph convolutional network with MLP-mixer based multi-path feature fusion for classification of histopathological imagesExpert Systems with Applications, 2022
- Hierarchical graph representations in digital pathologyMedical Image Analysis, 2022
- Deep and machine learning techniques for medical imaging-based breast cancer: A comprehensive reviewExpert Systems with Applications, 2021
- Classification of non-small cell lung cancer using one-dimensional convolutional neural networkExpert Systems with Applications, 2020
- Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networksMedical Image Analysis, 2020
- Deep learning for prediction of colorectal cancer outcome: a discovery and validation studyThe Lancet, 2020
- Multiple instance learning: A survey of problem characteristics and applicationsPattern Recognition, 2018
- Revisiting multiple instance neural networksPattern Recognition, 2018
- Mutational landscape and significance across 12 major cancer typesNature, 2013
- Solving the multiple instance problem with axis-parallel rectanglesArtificial Intelligence, 1997