A Rewiring Contrastive Patch PerformerMixer Framework for Graph Representation Learning
Authors:
- 15 December 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 5930-5939
Abstract
Integrating transformers with graph representation learning has emerged as a research focal point. However, recent studies showed that positional encoding in Transformers does not capture enough structural information between nodes. Additionally, existing graph neural network (GNN) models face the oversquashing issue, impeding information retention from distant nodes. To address, we transform graphs into regular structures, such as tokens, to enhance positional understanding and leverage transformer strengths. Inspired by the visual transformer (ViT) model, we propose partitioning graphs into patches and apply GNN models obtain fixed size vectors. Notably, our approach adopts contrastive learning for in-depth graph structure and incorporate more topological information via Ricci curvature to alleviate over-squashing problem by attenuating the effects of negatively curved edges while preserving the original graph structure. Unlike existing graph rewiring methods that directly modify graph structure by adding or removing edges, this approach is potentially more suitable for applications such as molecular learning where structural preservation is important. Our innovative pipeline subsequently introduces the PerformerMixer, a transformer variant with linear complexity, ensuring efficient computation. Evaluations on real-world benchmarks demonstrate our framework’s superior performance, like Peptides-func and achieve 3-WL expressiveness.Keywords
This publication has 30 references indexed in Scilit:
- MONEY: Ensemble learning for stock price movement prediction via a convolutional network with adversarial hypergraph modelAI Open, 2023
- Is Unimodal Bias Always Bad for Visual Question Answering? A Medical Domain Study with Dynamic Attention2022
- Contrastive self-supervised learning: review, progress, challenges and future research directionsInternational Journal of Multimedia Information Retrieval, 2022
- Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment ContrastJournal of Chemical Information and Modeling, 2022
- Molecular contrastive learning of representations via graph neural networksNature Machine Intelligence, 2022
- Relation-aware dynamic attributed graph attention network for stocks recommendationPattern Recognition, 2022
- Molecular generative Graph Neural Networks for Drug DiscoveryNeurocomputing, 2021
- Robust Audio-Visual Instance Discrimination2021
- Contrastive Representation Learning: A Framework and ReviewIEEE Access, 2020
- Momentum Contrast for Unsupervised Visual Representation Learning2020