I/O-Aware Batch Scheduling for Petascale Computing Systems
Authors:
- Zhou ZhouZZZhou Zhou
- Xu YangXYXu Yang
- Dongfang ZhaoDZDongfang Zhao
- Paul RichPRPaul Rich
- Wei TangWTWei Tang
- Jia WangJWJia Wang
- Zhiling LanZLZhiling Lan
- 1 September 2015
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 15525244,p. 254-263
Abstract
In the Big Data era, the gap between the storage performance and an application's I/O requirement is increasing. I/O congestion caused by concurrent storage accesses from multiple applications is inevitable and severely harms the performance. Conventional approaches either focus on optimizing an application's access pattern individually or handle I/O requests on a low-level storage layer without any knowledge from the upper-level applications. In this paper, we present a novel I/O-aware batch scheduling framework to coordinate ongoing I/O requests on petascale computing systems. The motivation behind this innovation is that the batch scheduler has a holistic view of both the system state and jobs' activities and can control the jobs' status on the fly during their execution. We treat a job's I/O requests as periodical subjobs within its lifecycle and transform the I/O congestion issue into a classical scheduling problem. We design two scheduling polices with different scheduling objectives either on user-oriented metrics or system performance. We conduct extensive trace-based simulations using real job traces and I/O traces from a production IBM Blue Gene/Q system. Experimental results demonstrate that our design can improve job performance by more than 30%, as well as increasing system performance.Keywords
This publication has 30 references indexed in Scilit:
- Towards Scalable Distributed Workload Manager with Monitoring-Based Weakly Consistent Resource Stealing2015
- Exploring the Design Tradeoffs for Extreme-Scale High-Performance Computing System SoftwareIEEE Transactions on Parallel and Distributed Systems, 2015
- Next generation job management systems for extreme-scale ensemble computing2014
- Reducing Energy Costs for IBM Blue Gene/P via Power-Aware Job Scheduling2014
- CALCioM: Mitigating I/O Interference in HPC Systems through Cross-Application Coordination2014
- Insights for exascale IO APIs from building a petascale IO API2013
- Characterizing output bottlenecks in a supercomputer2012
- On the role of burst buffers in leadership-class storage systems2012
- PLFS2009
- Scheduling for Parallel Supercomputing: A Historical Perspective of Achievable Utilization1999