Map-reduce-merge
Authors:
Open Access
- 11 June 2007
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1029-1040
Abstract
Map-Reduce is a programming model that enables easy development of scalable parallel applications to process a vast amount of data on large clusters of commodity machines. Through a simple interface with two functions, map and reduce, this model facilitates parallel implementation of many real-world tasks such as data processing jobs for search engines and machine learning. However,this model does not directly support processing multiple related heterogeneous datasets. While processing relational data is a common need, this limitation causes difficulties and/or inefficiency when Map-Reduce is applied on relational operations like joins. We improve Map-Reduce into a new model called Map-Reduce-Merge. It adds to Map-Reduce a Merge phase that can efficiently merge data already partitioned and sorted (or hashed) by map and reduce modules. We also demonstrate that this new model can express relational algebra operators as well as implement several join algorithms.Keywords
This publication has 5 references indexed in Scilit:
- Dryad2007
- Scientific data management in the coming decadeACM SIGMOD Record, 2005
- The Google file system2003
- Optimizing data aggregation for cluster-based internet services2003
- Parallel database systemsCommunications of the ACM, 1992