BLAD: Adaptive Load Balanced Scheduling and Operator Overlap Pipeline For Accelerating The Dynamic GNN Training
Open Access
- 11 November 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-13
Abstract
Dynamic graph networks are widely used for learning time-evolving graphs, but prior work on training these networks is inefficient due to communication overhead, long synchronization, and poor resource usage. Our investigation shows that communication and synchronization can be reduced by carefully scheduling the workload. And the execution order of operators in GNNs can be adjusted without hurting training convergence. We propose a system called BLAD to consider the above factors, comprising a two-level load scheduler and an overlap-aware topology manager. The scheduler allocates each snapshot group to a GPU, alleviating cross-GPU communication. The snapshots in a group are then carefully allocated to processes on a GPU, enabling overlap of compute-intensive NN operators and memory-intensive graph operators. The topology manager adjusts the operators' execution order to maximize the overlap. Experiments show that BLAD achieves 27.2% speed up on training time on average without affecting final accuracy, compared to state-of-the-art solutions.Keywords
Funding Information
- National Natural Science Foundation of China (62232011, 62022057, 61832006)
- Shanghai international science and technology collaboration project (21510713600)
This publication has 32 references indexed in Scilit:
- Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) networkPhysica D: Nonlinear Phenomena, 2020
- Dynamic graph convolutional networksPattern Recognition, 2020
- PipeDream2019
- Predicting Path Failure In Time-Evolving Graphs2019
- AliGraph: A Comprehensive Graph Neural Network Platform2019
- Community Interaction and Conflict on the Web2018
- NeurosurgeonACM SIGARCH Computer Architecture News, 2017
- Optimizing MPI Communication on Multi-GPU Systems Using CUDA Inter-Process Communication2012
- Graphs over time2005
- A Fast and High Quality Multilevel Scheme for Partitioning Irregular GraphsSIAM Journal on Scientific Computing, 1998