Invariant Graph Transformer
Authors:
- Zhe XuZXZhe Xu
- Menghai PanMPMenghai Pan
- Yuzhong ChenYCYuzhong Chen
- Huiyuan ChenHCHuiyuan Chen
- Yuchen YanYYYuchen Yan
- Mahashweta DasMDMahashweta Das
- Hanghang TongHTHanghang Tong
Preprint
Abstract
Rationale discovery is defined as finding a subset of the input data that maximally supports the prediction of downstream tasks. In graph machine learning context, graph rationale is defined to locate the critical subgraph in the given graph topology, which fundamentally determines the prediction results. In contrast to the rationale subgraph, the remaining subgraph is named the environment subgraph. Graph rationalization can enhance the model performance as the mapping between the graph rationale and prediction label is viewed as invariant, by assumption. To ensure the discriminative power of the extracted rationale subgraphs, a key technique named "intervention" is applied. The core idea of intervention is that given any changing environment subgraphs, the semantics from the rationale subgraph is invariant, which guarantees the correct prediction result. However, most, if not all, of the existing rationalization works on graph data develop their intervention strategies on the graph level, which is coarse-grained. In this paper, we propose well-tailored intervention strategies on graph data. Our idea is driven by the development of Transformer models, whose self-attention module provides rich interactions between input nodes. Based on the self-attention module, our proposed invariant graph Transformer (IGT) can achieve fine-grained, more specifically, node-level and virtual node-level intervention. Our comprehensive experiments involve 7 real-world datasets, and the proposed IGT shows significant performance advantages compared to 13 baseline methods.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Out-of-Distribution Generalization on Graphs: A SurveyIEEE Transactions on Pattern Analysis and Machine Intelligence, 2025
- Causal Machine Learning: A Survey and Open ProblemsFoundations and Trends® in Optimization, 2025
- Multimodal Learning With Transformers: A SurveyIEEE Transactions on Pattern Analysis and Machine Intelligence, 2023
- S u G e R: A Subgraph-based Graph Convolutional Network Method for Bundle Recommendation2022
- Graph Rationalization with Environment-based Augmentations2022
- OOD-GNN: Out-of-Distribution Generalized Graph Neural NetworkIEEE Transactions on Knowledge and Data Engineering, 2022
- A Survey on Vision TransformerIEEE Transactions on Pattern Analysis and Machine Intelligence, 2022
- A Comprehensive Survey on Graph Neural NetworksIEEE Transactions on Neural Networks and Learning Systems, 2020