Sparse Graph Transformer With Contrastive Learning
- 30 December 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computational Social Systems
- Vol. 11 (1) , 892-904
Abstract
Information aggregation and propagation over networks via graph neural networks (GNNs) plays an important role in node or graph representation learning, which currently depend on the calculation with a fixed adjacency matrix, facing over-smoothing problem, and difficulty to stack multiple layers for high-level representations. In contrast, Transformer calculates an importance score for each node to learn its embedding via the attention mechanism and has achieved great successes in many natural language processing (NLP) and computer vision (CV) tasks. However, Transformer is inflexible to extend to graphs, as its input and output must have the same dimension. It will also become intractable to allocate attention over a large-scale graph due to distractions. Moreover, most graph Transformers are trained in supervised ways, which consume additional resources to annotate samples with potentially wrong labels and have limited generalization of representations. Therefore, this article attempts to build a new Sparse Graph Transformer with Contrastive learning for graph representation learning, called SGTC. Specifically, we first employ centrality measures to remove the redundant topological information from input graph according to the influences of nodes and edges, then disturb the pruned graph to get two different augmentation views, and learn node representations in a contrastive manner. Besides, a novel sparse attention mechanism is also proposed to capture structural features of graphs, which effectively save memory and training time. SGTC can produce low-dimensional and high-order node representations, which have better generalization for multiple tasks. The proposed model is evaluated on three downstream tasks over six networks, and experimental results confirm its superior performance against the state-of-the-art baselines.Keywords
Funding Information
- National Natural Science Foundation of China (62076065)
- Natural Science Foundation of Fujian Province (2020J01495)
This publication has 18 references indexed in Scilit:
- GCC2020
- Graph deconvolutional networksInformation Sciences, 2020
- Heterogeneous Graph Transformer2020
- Graph Representation Learning via Graphical Mutual Information Maximization2020
- Joint sparse graph and flexible embedding for graph-based semi-supervised learningNeural Networks, 2019
- Network Embedding as Matrix Factorization2018
- struc2vec2017
- Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs2017
- node2vec2016
- DeepWalk2014