EC-Graph: A Distributed Graph Neural Network System with Error-Compensated Compression
Authors:
- 1 May 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 648-660
Abstract
The high training costs of graph neural networks (GNNs) have limited their applicability on large graphs, e.g., graphs with hundreds of millions of vertices which have become common in the era of big data. A few recent studies propose distributed GNN systems. However, these systems may generate high communication costs due to the extensive message passing among graph vertices stored on different machines. To address such limitations, in the paper, 1) we propose a distributed GNN computation system named EC-Graph for CPU clusters, which drastically reduces the communication costs among the machines by message compression; 2) we design a requesting-end compensation method for the embeddings to mitigate the errors induced by compression in the forward propagation and a Bit-Tuner to adaptively balance the model accuracy and message size; and 3) we propose a responding-end compensation approach for the embedding gradients in the backward propagation. Extensive experiments over large real-world datasets show that EC-Graph outperforms state-of-the-art distributed GNN systems on two CPU clusters of different sizes.Keywords
Funding Information
- National Natural Science Foundation of China (62072083,61902366)
This publication has 18 references indexed in Scilit:
- PSGraph: How Tencent trains extremely large-scale graphs with Spark?2020
- AliGraphProceedings of the VLDB Endowment, 2019
- PS22019
- FlexPSProceedings of the VLDB Endowment, 2018
- Hybrid Pulling/Pushing for I/O-Efficient Distributed and Iterative Graph Computing2016
- Petuum: A New Platform for Distributed Machine Learning on Big DataIEEE Transactions on Big Data, 2015
- SYNC or ASYNC: time to fuse for distributed graph-parallel computation2015
- Pregel2010
- Bandwidth optimal all-reduce algorithms for clusters of workstationsJournal of Parallel and Distributed Computing, 2009
- The Graph Neural Network ModelIEEE Transactions on Neural Networks, 2008