IOAgent: Democratizing Trustworthy HPC I/O Performance Diagnosis Capability via LLMs
Authors:
- 3 June 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 322-334
Abstract
As the complexity of the HPC storage stack rapidly grows, domain scientists face increasing challenges in effectively utilizing HPC storage systems to achieve their desired I/O performance. To identify and address I/O issues, scientists largely rely on I/O experts to analyze their I/O traces and provide insights into potential problems. However, with a limited number of I/O experts and the growing demand for dataintensive applications, inaccessibility has become a major bottleneck, hindering scientists from maximizing their productivity. The recent rapid progress in large language models (LLMs) opens the door to creating an automated tool that democratizes trustworthy I/O performance diagnosis capabilities to domain scientists. However, LLMs face significant challenges in this task, such as the inability to handle long context windows, a lack of accurate domain knowledge about HPC I/O, and the generation of hallucinations during complex interactions. In this work, we propose IOAgent as a systematic effort to address these challenges. IOAgent integrates various new designs, including a module-based pre-processor, a RAG-based domain knowledge integrator, and a tree-based merger to accurately diagnose I/O issues from a given Darshan trace file. Similar to an I/O expert, IOAgent provides detailed justifications and references for its diagnoses and offers an interactive interface for scientists to continue asking questions about the diagnosis. To evaluate IOAgent, we collected a diverse set of labeled job traces and released the first open diagnosis test suite, TraceBench. Based on this test suite, extensive evaluations were conducted, demonstrating that IOAgent matches or outperforms state-of-the-art I/O diagnosis tools with accurate and useful diagnosis results. We also show that IOAgent is not tied to specific LLMs, performing similarly well with both proprietary and open-source LLMs. We believe IOAgent has the potential to become a powerful tool for scientists navigating complex HPC I/O subsystems in the future.Keywords
All Related Versions
Funding Information
- National Science Foundation (CNS-2008265,CCF2412345,OAC-2417850,BCS-2416846)
This publication has 33 references indexed in Scilit:
- Can large language models reason about medical questions?Patterns, 2024
- Selective Propositional Reasoning: A Cognitive Load-Aware Strategy for Enhanced Reasoning2023
- Drill: Log-based Anomaly Detection for Large-scale Storage Systems Using Source Code Analysis2023
- ClusterLog: Clustering Logs for Effective Log-based Anomaly Detection2022
- Recorder 2.0: Efficient Parallel I/O Tracing and Analysis2020
- IOMiner: Large-Scale Analytics Framework for Gaining Knowledge from I/O Logs2018
- QMCPACK: an open sourceab initioquantum Monte Carlo package for the electronic structure of atoms, molecules and solidsJournal of Physics: Condensed Matter, 2018
- Log-Assisted Straggler-Aware I/O Scheduler for High-End Computing2016
- Nyx: A MASSIVELY PARALLEL AMR CODE FOR COMPUTATIONAL COSMOLOGYThe Astrophysical Journal, 2013
- Stack Trace Analysis for Large Scale Debugging2007