Analysis and Correlation of Application I/O Performance and System-Wide I/O Activity
Authors:
- Sandeep MadireddySMSandeep Madireddy
- Prasanna BalaprakashPBPrasanna Balaprakash
- Philip CarnsPCPhilip Carns
- Robert LathamRLRobert Latham
- Robert RossRRRobert Ross
- Shane SnyderSSShane Snyder
- Stefan M. WildSWStefan M. Wild
- 1 August 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-10
Abstract
Storage resources in high-performance computing are shared across all user applications. Consequently, storage performance can vary markedly, depending not only on an application's workload but also on what other activity is concurrently running across the system. This variability in storage performance is directly reflected in overall execution time variability, thus confounding efforts to predict job performance for scheduling or capacity planning. I/O variability also complicates the seemingly straightforward process of performance measurement when evaluating application optimizations. In this work we present a methodology to measure I/O contention with more rigor than in prior work. We apply statistical techniques to gain insight from application-level statistics and storage-side logging. We examine different correlation metrics for relating system workload to job I/O performance and identify an effective and generally applicable metric for measuring job I/O performance. We further demonstrate that the system-wide monitoring granularity can directly affect the strength of correlation observed. Insufficient granularity and measurements can hide the correlations between application I/O performance and system-wide I/O activity.Keywords
This publication has 17 references indexed in Scilit:
- CLARISSE: A Middleware for Data-Staging Coordination and Control on Large-Scale HPC Platforms2016
- On the Root Causes of Cross-Application I/O Interference in HPC Storage Systems2016
- Dynamic Model-Driven Parallel I/O Performance Tuning2015
- Collective I/O Tuning Using Analytical and Machine Learning Models2015
- CALCioM: Mitigating I/O Interference in HPC Systems through Cross-Application Coordination2014
- ALCF I/O Data Repository2013
- Managing Variability in the IO Performance of Petascale Storage Systems2010
- Parallel I/O performance: From events to ensembles2010
- Relative performance of mutual information estimation methods for quantifying the dependence among short and noisy dataPhysical Review E, 2007
- Random ForestsMachine Learning, 2001