Spark deployment and performance evaluation on the MareNostrum supercomputer
- 1 October 2015
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 299-306
Abstract
In this paper we present a framework to enable data-intensive Spark workloads on MareNostrum, a petascale supercomputer designed mainly for compute-intensive applications. As far as we know, this is the first attempt to investigate optimized deployment configurations of Spark on a petascale HPC setup. We detail the design of the framework and present some benchmark data to provide insights into the scalability of the system. We examine the impact of different configurations including parallelism, storage and networking alternatives, and we discuss several aspects in executing Big Data workloads on a computing system that is based on the compute-centric paradigm. Further, we derive conclusions aiming to pave the way towards systematic and optimized methodologies for fine-tuning data-intensive application on large clusters emphasizing on parallelism configurations.Keywords
This publication has 11 references indexed in Scilit:
- An architecture for compiling UDF-centric workflowsProceedings of the VLDB Endowment, 2015
- Hadoop superlinear scalabilityCommunications of the ACM, 2015
- Anti-combining for MapReduce2014
- A Tale of Two Data-Intensive Paradigms: Applications, Abstractions, and Architectures2014
- Characterization and Optimization of Memory-Resident MapReduce on HPC Systems2014
- Shark2013
- Cumulon2013
- Scalable k-means++Proceedings of the VLDB Endowment, 2012
- No one (cluster) size fits all2011
- Evaluation of HPC Applications on Cloud2011