A Cost-Efficient Auto-Scaling Algorithm for Large-Scale Graph Processing in Cloud Environments with Heterogeneous Resources
- 14 August 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Software Engineering
- Vol. 47 (8) , 1729-1741
Abstract
Graph processing model is being adopted extensively in various domains such as online gaming, social media, scientific computing and Internet of Things (IoT). Many frameworks have been developed in recent years to facilitate analytics and computing of large-scale graphs. Dynamic scalability is always a major concern and It becomes even more important when there is a correlation between scalability and monetary cost. The pay-as-you-go model that is used by public cloud providers enables users to pay only for the number of resources they utilize. Nevertheless, processing large-scale graphs in such environments has been less studied. In this paper, we have developed algorithms to take advantage of resource heterogeneity in cloud environments. Using these algorithms, the system can automatically adjust the number and types of virtual machines according to the computation requirements for convergent graph applications to improve the performance and reduce the dollar cost of the entire operation. Also, a smart profiling mechanism along with a novel dynamic repartitioning approach helps to distribute graph partitions expeditiously. It is shown that this method outperforms popular frameworks such as Giraph and decreases more than 50% of the dollar cost compared to Giraph.Keywords
Funding Information
- ARC Future Fellowship
- ARC Discovery
This publication has 31 references indexed in Scilit:
- PregelixProceedings of the VLDB Endowment, 2014
- LogGPProceedings of the VLDB Endowment, 2014
- X-Stream2013
- From "think like a vertex" to "think like a graph"Proceedings of the VLDB Endowment, 2013
- Presto2013
- Mizan2013
- Giraphx: Parallel Yet Serializable Large-Scale Graph Processing2013
- Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utilityFuture Generation Computer Systems, 2009
- A bridging model for parallel computationCommunications of the ACM, 1990
- Computing connected components on parallel computersCommunications of the ACM, 1979