A Parallel Distributed Weka Framework for Big Data Mining Using Spark
Authors:
- Aris-Kyriakos KoliopoulosAKAris-Kyriakos Koliopoulos
- Paraskevas YiapanisPYParaskevas Yiapanis
- Firat TekinerFTFirat Tekiner
- Goran NenadicGNGoran Nenadic
- John KeaneJKJohn Keane
- 1 June 2015
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 9-16
Abstract
Effective Big Data Mining requires scalable and efficient solutions that are also accessible to users of all levels of expertise. Despite this, many current efforts to provide effective knowledge extraction via large-scale Big Data Mining tools focus more on performance than on use and tuning which are complex problems even for experts. Weka is a popular and comprehensive Data Mining workbench with a well-known and intuitive interface, nonetheless it supports only sequential single-node execution. Hence, the size of the datasets and processing tasks that Weka can handle within its existing environment is limited both by the amount of memory in a single node and by sequential execution. This work discusses DistributedWekaSpark, a distributed framework for Weka which maintains its existing user interface. The framework is implemented on top of Spark, a Hadoop-related distributed framework with fast in-memory processing capabilities and support for iterative computations. By combining Weka's usability and Spark's processing power, DistributedWekaSpark provides a usable prototype distributed Big Data Mining workbench that achieves near-linear scaling in executing various real-world scale workloads - 91.4% weak scaling efficiency on average and up to 4x faster on average than Hadoop.Keywords
This publication has 13 references indexed in Scilit:
- RABID: A Distributed Parallel R for Large Datasets2014
- MLI: An API for Distributed Machine Learning2013
- Scale-up vs scale-out for Hadoop2013
- The family of mapreduce and large-scale data processing systemsACM Computing Surveys, 2013
- Toolkit-Based High-Performance Data Mining of Large Data on MapReduce Clusters2009
- The WEKA data mining softwareACM SIGKDD Explorations Newsletter, 2009
- MapReduceCommunications of the ACM, 2008
- Adapting the Weka Data Mining Toolkit to a Grid Based EnvironmentLecture Notes in Computer Science, 2005
- Efficient clustering of high-dimensional data sets with application to reference matching2000
- Parallel mining of association rulesIEEE Transactions on Knowledge and Data Engineering, 1996