Automatically Leveraging MapReduce Frameworks for Data-Intensive Applications
Authors:
- Maaz Bin Safeer AhmadMAMaaz Bin Safeer Ahmad
- Alvin CheungACAlvin Cheung
Preprint
Abstract
MapReduce is a popular programming paradigm for developing large-scale, data-intensive computation. Many frameworks that implement this paradigm have recently been developed. To leverage these frameworks, however, developers must become familiar with their APIs and rewrite existing code. Casper is a new tool that automatically translates sequential Java programs into the MapReduce paradigm. Casper identifies potential code fragments to rewrite and translates them in two steps: (1) Casper uses program synthesis to search for a program summary (i.e., a functional specification) of each code fragment. The summary is expressed using a high-level intermediate language resembling the MapReduce paradigm and verified to be semantically equivalent to the original using a theorem prover. (2) Casper generates executable code from the summary, using either the Hadoop, Spark, or Flink API. We evaluated Casper by automatically converting real-world, sequential Java benchmarks to MapReduce. The resulting benchmarks perform up to 48.2x faster compared to the original.Keywords
All Related Versions
- Version 1, 2018-01-30, ArXiv
- Version 2, 2018-06-19, ArXiv
- Published version: , 1205.
This publication has 7 references indexed in Scilit:
- The Commutativity Problem of the MapReduce Framework: A Transducer-Based Approach2016
- Array Folds Logic2016
- Implicit Parallelism through Deep Language EmbeddingACM SIGMOD Record, 2016
- Algorithmic program synthesis: introductionInternational Journal on Software Tools for Technology Transfer, 2013
- Fast nonlocal filtering applied to electron cryomicroscopy2008
- Automatic Detection of Parallelism: A grand challenge for high performance computingIEEE Parallel & Distributed Technology: Systems & Applications, 1994
- An axiomatic basis for computer programmingCommunications of the ACM, 1969