Enabling Big Data Analytics in the Hybrid Cloud Using Iterative MapReduce
- 1 December 2015
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 290-299
Abstract
The cloud computing model has seen tremendous commercial success through its materialization via two prominent models to date, namely public and private cloud. Recently, a third model combining the former two service models as on-/off-premise resources has been receiving significant market traction: hybrid cloud. While state of art techniques that address workload performance prediction and efficient workload execution over hybrid cloud setups exist, how to address data-intensive workloads - including Big Data Analytics - in similar environments is nascent. This paper addresses this gap by taking on the challenge of bursting over hybrid clouds for the benefit of accelerating iterative MapReduce applications. We first specify the challenges associated with data locality and data movement in such setups. Subsequently, we propose a novel technique to address the locality issue, without requiring changes to the MapReduce framework or the underlying storage layer. In addition, we contribute with a performance prediction methodology that combines modeling with micro-benchmarks to estimate completion time for iterative MapReduce applications, which enables users to estimate cost-to-solution before committing extra resources from public clouds. We show through experimentation in a dual-Openstack hybrid cloud setup that our solutions manage to bring substantial improvement at predictable cost-control for two real-life iterative MapReduce applications: large-scale machine learning and text analysis.Keywords
This publication has 24 references indexed in Scilit:
- Bursting the Cloud Data Bubble: Towards Transparent Storage Elasticity in IaaS Clouds2014
- Accurate Resource Prediction for Hybrid IaaS Clouds Using Workload-Tailored Elastic Compute Units2013
- HybridMR: A Hierarchical MapReduce Scheduler for Hybrid Data Centers2013
- Scaling MapReduce Applications Across Hybrid Clouds to Meet Soft Deadlines2013
- Failure-aware resource provisioning for hybrid Cloud infrastructureJournal of Parallel and Distributed Computing, 2012
- A Framework for Data-Intensive Computing with Cloud Bursting2011
- A capabilities-aware framework for using computational accelerators in data-intensive computingJournal of Parallel and Distributed Computing, 2011
- HadoopDBProceedings of the VLDB Endowment, 2009
- MapReduceCommunications of the ACM, 2008
- Clustering Methods: A History of k-Means Algorithms2007