Large Scale Distributed Data Science using Apache Spark
Authors:
Open Access
- 10 August 2015
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 2323-2324
Abstract
Apache Spark is an open-source cluster computing framework for big data processing. It has emerged as the next generation big data processing engine, overtaking Hadoop MapReduce which helped ignite the big data revolution. Spark maintains MapReduce's linear scalability and fault tolerance, but extends it in a few important ways: it is much faster (100 times faster for certain applications), much easier to program in due to its rich APIs in Python, Java, Scala (and shortly R), and its core data abstraction, the distributed data frame, and it goes far beyond batch applications to support a variety of compute-intensive tasks, including interactive queries, streaming, machine learning, and graph processing. This tutorial will provide an accessible introduction to Spark and its potential to revolutionize academic and commercial data science practices.Keywords
This publication has 2 references indexed in Scilit:
- PLANETProceedings of the VLDB Endowment, 2009
- Matrix factorization and neighbor based algorithms for the netflix prize problem2008