ApproxHadoop
Open Access
- 14 March 2015
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 383-397
Abstract
We propose and evaluate a framework for creating and running approximation-enabled MapReduce programs. Specifically, we propose approximation mechanisms that fit naturally into the MapReduce paradigm, including input data sampling, task dropping, and accepting and running a precise and a user-defined approximate version of the MapReduce code. We then show how to leverage statistical theories to compute error bounds for popular classes of MapReduce programs when approximating with input data sampling and/or task dropping. We implement the proposed mechanisms and error bound estimations in a prototype system called ApproxHadoop. Our evaluation uses MapReduce applications from different domains, including data analytics, scientific computing, video encoding, and machine learning. Our results show that ApproxHadoop can significantly reduce application execution time and/or energy consumption when the user is willing to tolerate small errors. For example, ApproxHadoop can reduce runtimes by up to 32x when the user can tolerate an error of 1% with 95% confidence. We conclude that our framework and system can make approximation easily accessible to many application domains using the MapReduce model.Keywords
Funding Information
- NSF (CSR-1117368)
This publication has 28 references indexed in Scilit:
- Statistical Methods for Estimating the Minimum Thickness Along a PipelineTechnometrics, 2015
- Expressing and verifying probabilistic assertions2014
- SAGE2013
- BlinkDB2013
- PARMA2012
- Managing performance vs. accuracy trade-offs with loop perforation2011
- Probabilistically Accurate Program Transformations2011
- Embedded Probabilistic Programming2009
- Optimized stratified sampling for approximate query processingACM Transactions on Database Systems, 2007
- Imprecise computationsProceedings of the IEEE, 1994