Graph-Aware and Sequence-Aware Multimodal Deep Learning Framework for Cancer Detection and Risk Analysis from Medical Imaging

Authors:
Preprint
Publisher website
Google Scholar
Abstract
Early and accurate cancer detection from medical imaging remains challenging because clinically relevant evidence is distributed across local image appearance, structural relationships between suspicious regions, and ordered imaging context. This study proposes a graph-aware sequence-aware deep learning framework for cancer analysis from medical imaging, combining a convolutional neural network (CNN) backbone for spatial feature extraction, a graph attention network (GAT) for lesion-structure modelling, and a bidirectional long short-term memory (BiLSTM) module for ordered-view or slice-sequence representation learning. Multimodal fusion is evaluated in a leakage-controlled setting only for the RSNA mammography task, where the metadata branch is restricted to patient age and implant status, both available before diagnosis. In contrast, the primary LIDC-IDRI experiment is conducted as an image-only analysis because radiologist malignancy scores and semantic nodule attributes are annotation-derived variables and are not treated as independent clinical predictors. The framework is evaluated on the RSNA Breast Cancer Detection dataset and the LIDC-IDRI lung CT dataset using accuracy, precision, recall, F1-score, and area under the receiver operating characteristic curve (AUC). Additional ablation experiments assess the contribution of graph learning, sequence-aware modelling, and leakage-safe metadata fusion, while SHAP analysis is used to quantify the influence of the included RSNA metadata variables on the final multimodal predictions. The proposed framework provides an interpretable and leakage-aware architecture for integrating spatial, relational, and ordered imaging information, and demonstrates how limited pre-diagnostic metadata can be incorporated without overstating multimodal novelty or clinical realism.

This publication has 13 references indexed in Scilit: