Sylvie: 3D-Adaptive and Universal System for Large-Scale Graph Neural Network Training
Authors:
- Haozhao WangHWHaozhao Wang
- Yonggang WenYWYonggang Wen
- Tianwei ZhangTZTianwei Zhang
- 13 May 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3823-3836
Abstract
Distributed full-graph training of Graph Neural Networks (GNNs) has been widely adopted to learn large-scale graphs. While recent system advancements can improve the training throughput of GNNs, their practical adoption is limited by the potential accuracy decline. This concern is particularly prominent in deeper and more intricate GNN architectures, where noticeable performance degradation becomes apparent. Moreover, existing works fail to comprehensively consider diverse opportunities for acceleration. Motivated by these deficiencies, we propose Sylvie,a full-graph training system that not only improves the training throughput substantially but also maintains the model quality for universal GNNs. By harnessing the inherent information embedded in the graph data and model structure, Sylvie intelligently optimizes GNN training across three key dimensions: data, time, and execution. It identifies performance-relevant features of the input graph offline as subsequent optimization guidance. Subsequently, Sylvie devises an online convergence-maintenance strategy that adaptively integrates and aligns GNN-specific quantization and inter-epoch asynchronous training with the real-time training characteristics. Extensive experiments demonstrate that Sylvie surpasses existing GNN training systems by up to 17.2× speedup for both shallow and deep GNNs, without compromising the model accuracy.Keywords
This publication has 32 references indexed in Scilit:
- ICS-GNN$^+$: lightweight interactive community search via graph neural networkThe VLDB Journal, 2022
- HybridGNN: Learning Hybrid Representation for Recommendation in Multiplex Heterogeneous Networks2022
- EC-Graph: A Distributed Graph Neural Network System with Error-Compensated Compression2022
- Accurate and Scalable Graph Neural Networks for Billion-Scale Graphs2022
- Dynamic Hypergraph Convolutional Network2022
- Adaptive Quantization of Model Updates for Communication-Efficient Federated Learning2021
- DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs2020
- Towards Unified INT8 Training for Convolutional Neural Network2020
- Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference2018
- Clustering and community detection in directed networks: A surveyPhysics Reports, 2013