Meta-MapReduce: A Technique for Reducing Communication in MapReduce Computations
Preprint
Abstract
MapReduce has proven to be one of the most useful paradigms in the revolution of distributed computing, where cloud services and cluster computing become the standard venue for computing. The federation of cloud and big data activities is the next challenge where MapReduce should be modified to avoid (big) data migration across remote (cloud) sites. This is exactly our scope of research, where only the very essential data for obtaining the result is transmitted, reducing communication, processing and preserving data privacy as much as possible. In this work, we propose an algorithmic technique for MapReduce algorithms, called Meta-MapReduce, that decreases the communication cost by allowing us to process and move metadata to clouds and from the map phase to reduce phase. In Meta-MapReduce, the reduce phase fetches only the required data at required iterations, which in turn, assists in preserving the data privacy.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- A Survey on Geographically Distributed Big-Data Processing Using MapReduceIEEE Transactions on Big Data, 2017
- End-to-End Optimization for Geo-Distributed MapReduceIEEE Transactions on Cloud Computing, 2014
- ComMapReduce: An improvement of MapReduce with lightweight communication mechanismsData & Knowledge Engineering, 2013
- G-Hadoop: MapReduce across distributed data centers for data-intensive computingFuture Generation Computer Systems, 2013
- Locality-aware dynamic VM reconfiguration on MapReduce clouds2012
- Hierarchical MapReduce Programming Model and Scheduling Algorithms2012
- Efficient parallel kNN joins for large data in MapReduce2012
- Purlieus2011
- Optimizing Multiway Joins in a Map-Reduce EnvironmentIEEE Transactions on Knowledge and Data Engineering, 2011
- Pregel2010