Hadoop Workloads Characterization for Performance and Energy Efficiency Optimizations on Microservers
- 5 September 2017
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multi-Scale Computing Systems
- Vol. 4 (3) , 355-368
Abstract
The traditional low-power embedded processors such as Atom are entering into the high-performance server market. At the same time, big data analytics applications are emerging and dramatically changing the landscape of data center workloads. Emerging big data applications require a significant amount of server computational power. However, the rapid growth in the data yields challenges to process them efficiently using current high-performance server architectures. Furthermore, physical design constraints, such as power and density have become the dominant limiting factor for scaling out servers. Numerous big data applications rely on using Hadoop MapReduce framework to perform their analysis on large-scale datasets. Since Hadoop configuration parameters and system parameters directly affect the MapReduce job performance and energy-efficiency, joint application, system and architecture level parameters tuning is vital to maximize the energy efficiency for Hadoop-based applications. In this work, through methodical investigation of performance and power measurements, we demonstrate how the interplay among various Hadoop configuration parameters, as well as system and architecture level parameters affect not only the performance but also the energy-efficiency across various big data applications. Our results identify trends to guide scheduling decision and key insights to help improving Hadoop MapReduce applications performance, power and energy-efficiency on microservers.Keywords
Funding Information
- US National Science Foundation (CNS 1526913)
This publication has 9 references indexed in Scilit:
- HPC node performance and energy modeling with the co-location of applicationsThe Journal of Supercomputing, 2016
- A performance study of big data on small nodesProceedings of the VLDB Endowment, 2015
- The Datacenter as a Computer2013
- CloudRank-D: benchmarking and ranking cloud computing systems for data processing applicationsFrontiers of Computer Science, 2012
- Investigation of Data Locality in MapReduce2012
- Energy efficient scheduling of MapReduce workloads on heterogeneous clusters2011
- Bubble-Up2011
- CloudCmp2010
- A survey of design techniques for system-level dynamic power managementIEEE Transactions on Very Large Scale Integration (VLSI) Systems, 2000