Challenges and Proposals for Enabling Dynamic Heterogeneous Execution of Big Data Frameworks
Authors:
- Maria XekalakiMXMaria Xekalaki
- Juan FumeroJFJuan Fumero
- Christos KotselidisCKChristos Kotselidis
- 1 December 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 335-341
Abstract
The efficient execution of Big Data applications requires a large quantity of compute and memory resources. Typically, these resources are in the form of data centres with numerous processing elements connected through a computer network. Although initially the majority of data centers were utilizing only CPU resources, nowadays we can find heterogeneous accelerators such as GPUs and FPGAs. Ideally, Big Data frameworks and applications should exploit those diverse hardware resources in order to push their performance boundaries or increase resource utilization. Despite ongoing work to enable such functionality, the majority of the solutions revolve around external libraries that provide pre-compiled kernels for heterogeneous accelerators. This fact imposes programmability and code fragmentation challenges that can only be addressed by enabling Big Data platforms to dynamically compile and execute their code on such devices. In this paper we analyze and discuss the major challenges for programming and executing Big Data processing applications on distributed systems with heterogeneous hardware. In addition, we present our work-in-progress towards providing a heterogeneous programming framework for running Big Data applications on systems that include diverse hardware resources including CPUs, GPUs, and FPGAs. In contrast to existing approaches, our envisioned solution employs JIT compilation and runtime support, integrated in the data flow engine, enabling the automatic acceleration of Big Data platforms completely transparently to the user and without sacrificing programmability.Keywords
This publication has 11 references indexed in Scilit:
- Heterogeneous Managed Runtime Systems2017
- Critical analysis of Big Data challenges and analytical methodsJournal of Business Research, 2017
- Voodoo - a vector algebra for portable database performance on modern hardwareProceedings of the VLDB Endowment, 2016
- Benchmarking Streaming Computation Engines: Storm, Flink and Spark Streaming2016
- Runtime Code Generation and Data Management for Heterogeneous Computing in Java2015
- HeteroDoop2015
- Glasswing2014
- A Composable Array Function Interface for Heterogeneous Computing in Java2014
- HadoopCL: MapReduce on Distributed Heterogeneous Platforms through Seamless Integration of Hadoop and OpenCL2013
- MapReduceCommunications of the ACM, 2008