The I/O Trace Initiative: Building a Collaborative I/O Archive to Advance HPC
Open Access
- 12 November 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1216-1222
Abstract
HPC application developers and administrators need to understand the complex interplay between compute clusters and storage systems to make effective optimization decisions. Ad hoc investigations of this interplay based on isolated case studies can lead to conclusions that are incorrect or difficult to generalize. The I/O Trace Initiative aims to improve the scientific community’s understanding of I/O operations by building a searchable collaborative archive of I/O traces from a wide range of applications and machines, with a focus on high-performance computing and scalable AI/ML. This initiative advances the accessibility of I/O trace data by enabling users to locate and compare traces based on user-specified criteria. It also provides a visual analytics platform for in-depth analysis, paving the way for the development of advanced performance optimization techniques. By acting as a hub for trace data, the initiative fosters collaborative research by encouraging data sharing and collective learning.Keywords
This publication has 12 references indexed in Scilit:
- tf-Darshan: Understanding Fine-grained I/O Performance in Machine Learning Workloads2020
- Recorder 2.0: Efficient Parallel I/O Tracing and Analysis2020
- Revisiting I/O behavior in large-scale storage systems2019
- Exascale Deep Learning for Climate Analytics2018
- Toward Understanding I/O Behavior in HPC Workflows2018
- IOMiner: Large-Scale Analytics Framework for Gaining Knowledge from I/O Logs2018
- Characterizing the performance benefit of hybrid memory system for HPC applicationsParallel Computing, 2018
- Modular HPC I/O Characterization with Darshan2016
- A multi-level approach for understanding I/O activity in HPC applications2013
- Understanding and Improving Computational Science Storage Access through Continuous CharacterizationACM Transactions on Storage, 2011