Performance and Monetary Cost of Large-Scale Distributed Graph Processing on Amazon Cloud
Authors:
- 20 October 2016
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 9-16
Abstract
Graph analytics has become essential to uncover relationship insights in complex systems. As graphs grow in scale, several graph-parallel frameworks including Pregel, GraphLab, and PowerGraph are developed based on commodity computers and/or Cloud instances. According to recent research and empirical performance evaluation, system optimization on PowerGraph allow it to outperform others significantly for processing natural graphs with skewed degree distribution. However, the performance characters, resource usage pattern and monetary cost have never been explored in-depth. In this paper, PowerGraph are evaluated with three different algorithms on Amazon EC2 instances with upto 768 CPU cores. We find that thegraph processing performance does not always increase with the increasing Cloud resources. Due to synchronization overheads, resources are not fully utilized. Graph processing tasks may prefer different execution strategies with specified number and type of Cloud instances to achieve high cost-efficiency, playing the trade-off between monetary cost and execution performance.Keywords
This publication has 15 references indexed in Scilit:
- An experimental comparison of pregel-like graph processing systemsProceedings of the VLDB Endowment, 2014
- How Well Do Graph-Processing Platforms Perform? An Empirical Performance Evaluation and Analysis2014
- GPS2013
- GraphX2013
- Improving large graph processing on partitioned graphs in the cloud2012
- Distributed GraphLabProceedings of the VLDB Endowment, 2012
- Design patterns for efficient graph algorithms in MapReduce2010
- The Hadoop Distributed File System2010
- On compressing social networks2009
- A bridging model for parallel computationCommunications of the ACM, 1990