SystemML: Declarative machine learning on MapReduce
Authors:
- 1 April 2011
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 10636382,p. 231-242
Abstract
MapReduce is emerging as a generic parallel programming paradigm for large clusters of machines. This trend combined with the growing need to run machine learning (ML) algorithms on massive datasets has led to an increased interest in implementing ML algorithms on MapReduce. However, the cost of implementing a large class of ML algorithms as low-level MapReduce jobs on varying data and machine cluster sizes can be prohibitive. In this paper, we propose SystemML in which ML algorithms are expressed in a higher-level language and are compiled and executed in a MapReduce environment. This higher-level language exposes several constructs including linear algebra primitives that constitute key building blocks for a broad class of supervised and unsupervised ML algorithms. The algorithms expressed in SystemML are compiled and optimized into a set of MapReduce jobs that can run on a cluster of machines. We describe and empirically evaluate a number of optimization strategies for efficiently executing these algorithms on Hadoop, an open-source MapReduce implementation. We report an extensive performance evaluation on three ML algorithms on varying data and cluster sizes.Keywords
This publication has 12 references indexed in Scilit:
- Distributed nonnegative matrix factorization for web-scale dyadic data analysis on mapreduce2010
- Boom analytics2010
- Parallel algorithms for mining large-scale rich-media data2009
- PLANETProceedings of the VLDB Endowment, 2009
- Building a high-level dataflow system on top of Map-ReduceProceedings of the VLDB Endowment, 2009
- SCOPEProceedings of the VLDB Endowment, 2008
- Fast support vector machine training and classification on graphics processors2008
- MapReduceCommunications of the ACM, 2008
- Google news personalization2007
- Evaluating MapReduce for Multi-core and Multiprocessor Systems2007