A Comprehensive Survey on Distributed Training of Graph Neural Networks
Preprint
Abstract
Graph neural networks (GNNs) have been demonstrated to be a powerful algorithmic model in broad application fields for their effectiveness in learning over graphs. To scale GNN training up for large-scale and ever-growing graphs, the most promising solution is distributed training which distributes the workload of training across multiple computing nodes. At present, the volume of related research on distributed GNN training is exceptionally vast, accompanied by an extraordinarily rapid pace of publication. Moreover, the approaches reported in these studies exhibit significant divergence. This situation poses a considerable challenge for newcomers, hindering their ability to grasp a comprehensive understanding of the workflows, computational patterns, communication strategies, and optimization techniques employed in distributed GNN training. As a result, there is a pressing need for a survey to provide correct recognition, analysis, and comparisons in this field. In this paper, we provide a comprehensive survey of distributed GNN training by investigating various optimization techniques used in distributed GNN training. First, distributed GNN training is classified into several categories according to their workflows. In addition, their computational patterns and communication patterns, as well as the optimization techniques proposed by recent work are introduced. Second, the software frameworks and hardware platforms of distributed GNN training are also introduced for a deeper understanding. Third, distributed GNN training is compared with distributed training of deep neural networks, emphasizing the uniqueness of distributed GNN training. Finally, interesting issues and opportunities in this field are discussed.Keywords
All Related Versions
This publication has 65 references indexed in Scilit:
- 2PGraph: Accelerating GNN Training over Large Graphs on GPU Clusters2021
- A graph placement methodology for fast chip designNature, 2021
- GCN-RL Circuit Designer: Transferable Transistor Sizing with Graph Neural Networks and Reinforcement Learning2020
- A Hitchhiker’s Guide On Distributed Training Of Deep Neural NetworksJournal of Parallel and Distributed Computing, 2020
- Graph neural networks: A review of methods and applicationsAI Open, 2020
- Survey of external memory large-scale graph processing on a multi-core systemThe Journal of Supercomputing, 2019
- Large-Scale Machine Learning with Stochastic Gradient Descent2010
- Community Structure in Large Networks: Natural Cluster Sizes and the Absence of Large Well-Defined ClustersInternet Mathematics, 2009
- Realistic, Mathematically Tractable Graph Generation and Evolution, Using Kronecker Multiplication2005
- Gene Ontology: tool for the unification of biologyNature Genetics, 2000