Simplifying Distributed Neural Network Training on Massive Graphs: Randomized Partitions Improve Model Aggregation
Open Access
- 28 December 2024
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Knowledge Discovery From Data
- Vol. 19 (1) , 1-26
Abstract
Distributed graph neural network (GNN) training facilitates learning on massive graphs that surpass the storage and computational capabilities of a single machine. Traditional distributed frameworks strive for performance parity with centralized training by maximally recovering cross-instance node dependencies, relying either on inter-instance communication or periodic fallback to centralized training. However, these processes create overhead and constrain the scalability of the framework. In this work, we propose a streamlined framework for distributed GNN training that eliminates these costly operations, yielding improved scalability, convergence speed, and performance over state-of-the-art approaches. Our framework (1) comprises independent trainers that asynchronously learn local models from locally available parts of the training graph and (2) synchronizes these local models only through periodic (time-based) model aggregation. Contrary to prevailing belief, our theoretical analysis shows that it is not essential to maximize the recovery of cross-instance node dependencies to achieve performance parity with centralized training. Instead, our framework leverages randomized assignment of nodes or super-nodes (i.e., collections of original nodes) to partition the training graph in order to enhance data uniformity and minimize discrepancies in gradient and loss function across instances. Experiments on social and e-commerce networks with up to 1.3 billion edges show that our proposed framework achieves state-of-the-art performance and 2.31 \(\times\) speedup compared to the fastest baseline despite using less training data.
Keywords
All Related Versions
Funding Information
- National Science Foundation (IIS 2212143)
This publication has 23 references indexed in Scilit:
- Graph Coarsening via Convolution Matching for Scalable Graph Neural Network Training2024
- Condensing Graphs via One-Step Gradient Matching2022
- DistGNN2021
- Microsoft Academic Graph: When experts are not enoughQuantitative Science Studies, 2020
- Cluster-GCN2019
- Graph Convolutional Neural Networks for Web-Scale Recommender Systems2018
- Modeling Relational Data with Graph Convolutional Networks2018
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- A Fast and High Quality Multilevel Scheme for Partitioning Irregular GraphsSIAM Journal on Scientific Computing, 1998
- Acceleration of Stochastic Approximation by AveragingSIAM Journal on Control and Optimization, 1992