A Comprehensive Survey on Distributed Training of Graph Neural Networks
Top Cited Papers
- 8 December 2023
- journal article
- review article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in Proceedings of the IEEE
- Vol. 111 (12) , 1572-1606
Abstract
Graph neural networks (GNNs) have been demonstrated to be a powerful algorithmic model in broad application fields for their effectiveness in learning over graphs. To scale GNN training up for large-scale and ever-growing graphs, the most promising solution is distributed training that distributes the workload of training across multiple computing nodes. At present, the volume of related research on distributed GNN training is exceptionally vast, accompanied by an extraordinarily rapid pace of publication. Moreover, the approaches reported in these studies exhibit significant divergence. This situation poses a considerable challenge for newcomers, hindering their ability to grasp a comprehensive understanding of the workflows, computational patterns, communication strategies, and optimization techniques employed in distributed GNN training. As a result, there is a pressing need for a survey to provide correct recognition, analysis, and comparisons in this field. In this article, we provide a comprehensive survey of distributed GNN training by investigating various optimization techniques used in distributed GNN training. First, distributed GNN training is classified into several categories according to their workflows. In addition, their computational patterns and communication patterns, as well as the optimization techniques proposed by recent work, are introduced. Second, the software frameworks and hardware platforms of distributed GNN training are also introduced for a deeper understanding. Third, distributed GNN training is compared with distributed training of deep neural networks (DNNs), emphasizing the uniqueness of distributed GNN training. Finally, interesting issues and opportunities in this field are discussed.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (61732018, 61872335, 62202451)
- Austrian-Chinese Cooperative Research and Development Project [Austrian Research Promotion Agency (FFG) and Chinese Academy of Sciences (CAS)] (171111KYSB20200002)
- CAS Project for Young Scientists in Basic Research (YSBR-029)
- CAS Project for Youth Innovation Promotion Association
This publication has 82 references indexed in Scilit:
- Deep learningNature, 2015
- Scalable matrix computations on large scale-free graphs using 2D graph partitioning2013
- Streaming graph partitioning for large distributed graphs2012
- Understanding throughput-oriented architecturesCommunications of the ACM, 2010
- Community Structure in Large Networks: Natural Cluster Sizes and the Absence of Large Well-Defined ClustersInternet Mathematics, 2009
- MapReduceCommunications of the ACM, 2008
- Realistic, Mathematically Tractable Graph Generation and Evolution, Using Kronecker Multiplication2005
- A Fast and High Quality Multilevel Scheme for Partitioning Irregular GraphsSIAM Journal on Scientific Computing, 1998
- Long Short-Term MemoryNeural Computation, 1997
- WordNetCommunications of the ACM, 1995