A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinking
Preprint
Abstract
Large-scale graph training is a notoriously challenging problem for graph neural networks (GNNs). Due to the nature of evolving graph structures into the training process, vanilla GNNs usually fail to scale up, limited by the GPU memory space. Up to now, though numerous scalable GNN architectures have been proposed, we still lack a comprehensive survey and fair benchmark of this reservoir to find the rationale for designing scalable GNNs. To this end, we first systematically formulate the representative methods of large-scale graph training into several branches and further establish a fair and consistent benchmark for them by a greedy hyperparameter searching. In addition, regarding efficiency, we theoretically evaluate the time and space complexity of various branches and empirically compare them w.r.t GPU memory usage, throughput, and convergence. Furthermore, We analyze the pros and cons for various branches of scalable GNNs and then present a new ensembling training manner, named EnGCN, to address the existing issues. Our code is available at https://github.com/VITA-Group/Large_Scale_GCN_Benchmarking.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Graph Attention Multi-Layer Perceptron2022
- Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark StudyIEEE Transactions on Pattern Analysis and Machine Intelligence, 2022
- Neighbor2Seq: Deep Learning on Massive Graphs by Transforming Neighbors to Sequences2022
- Combining Graph Convolutional Neural Networks and Label PropagationACM Transactions on Information Systems, 2021
- Implementation of the article "DistGNN: scalable distributed training for large-scale graph neural networks"2021
- Dynamic Graph CNN for Learning on Point CloudsACM Transactions on Graphics, 2019
- Predicting multicellular function through multi-layer tissue networksBioinformatics, 2017
- Label Propagation through Linear NeighborhoodsIEEE Transactions on Knowledge and Data Engineering, 2007
- Comparison of descriptor spaces for chemical compound retrieval and classificationKnowledge and Information Systems, 2007
- A Fast and High Quality Multilevel Scheme for Partitioning Irregular GraphsSIAM Journal on Scientific Computing, 1998