New YARN Non-Exclusive Resource Management Scheme through Opportunistic Idle Resource Assignment
- 29 August 2018
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cloud Computing
- Vol. 9 (2) , 696-709
Abstract
Efficiently managing resources and improving throughput in a large-scale cluster has become a crucial problem with the explosion of data processing applications in recent years. Hadoop YARN and Mesos, as two universal resource manage- ment platforms, have been widely adopted in the commodity cluster for co-deploying multiple data processing frameworks, such as Hadoop MapReduce and Apache Spark. However, in the existing resource management, a certain amount of resources are exclusively allocated to a running task and can only be re-assigned after that task is completed. This exclusive mode unfortunately leads to a potential problem that may under-utilize the cluster resources and degrade system performance. To address this issue, we propose a novel opportunistic and efficient resource allocation scheme, named O P ERA, which breaks the barriers among the encapsulated resource containers by leveraging the knowledge of actual runtime resource utilizations to re-assign opportunistic available resources to the pending tasks. O P ERA avoids incurring severe performance interference to active tasks by further using two approaches to efficiently balances the starvations of reserved tasks and normal queued tasks. We implement and evaluate O P ERA in Hadoop YARN v2.5.Keywords
Funding Information
- National Science Foundation Career (CNS-1452751)
- National Science Foundation (CNS-1552525)
- Air Force Office of Scientific Research (FA9550-14-1-0160)
This publication has 18 references indexed in Scilit:
- MR-SPS: Scalable parallel scheduler for YARN/MapReduce platform2015
- MapReduce Scheduling for Deadline-Constrained Jobs in Heterogeneous Cloud Computing SystemsIEEE Transactions on Cloud Computing, 2015
- Large-scale cluster management at Google with Borg2015
- Self-Adjusting Slot Configurations for Homogeneous and Heterogeneous Hadoop ClustersIEEE Transactions on Cloud Computing, 2015
- FRESH: Fair and Efficient Slot Configuration and Scheduling for Hadoop Clusters2014
- HaSTE: Hadoop YARN Scheduling Based on Task-Dependency and Resource-Demand2014
- DynMR2014
- Quasar2014
- Apache Hadoop YARN2013
- MapReduceCommunications of the ACM, 2008