I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey
Open Access
- 7 May 2025
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Computing Surveys
- Vol. 57 (10) , 1-41
Abstract
Growing interest in Artificial Intelligence (AI) has resulted in a surge in demand for faster methods of Machine Learning (ML) model training and inference. This demand for speed has prompted the use of high performance computing (HPC) systems that excel in managing distributed workloads. Because data is the main fuel for AI applications, the performance of the storage and I/O subsystem of HPC systems is critical. In the past, HPC applications accessed large portions of data written by simulations or experiments or ingested data for visualizations or analysis tasks. ML workloads perform small reads spread across a large number of random files. This shift of I/O access patterns poses several challenges to modern parallel storage systems. In this article, we survey I/O in ML applications on HPC systems, and target literature within a 6-year time window from 2019 to 2024. We define the scope of the survey, provide an overview of the common phases of ML, review available profilers and benchmarks, examine the I/O patterns encountered during offline data preparation, training, and inference, and explore I/O optimizations utilized in modern ML frameworks and proposed in recent literature. Lastly, we seek to expose research gaps that could spawn further R&D.Keywords
All Related Versions
Funding Information
- U.S. Department of Energy
- Office of Science, Office of Advanced Scientific Computing Research (DE-AC02-05CH11231)
- National Energy Research Scientific Computing Center
- Office of Science of the U.S. Department of Energy (DE-AC02-05CH11231)
This publication has 95 references indexed in Scilit:
- Asynchronous federated learning on heterogeneous devices: A surveyComputer Science Review, 2023
- An efficient algorithm for data parallelism based on stochastic optimizationAlexandria Engineering Journal, 2022
- Understanding and Leveraging the I/O Patterns of Emerging Machine Learning Analytics2022
- Analyzing the I/O Patterns of Deep Learning Applications2021
- ExaHDF5: Delivering Efficient Parallel I/O on Exascale Computing SystemsJournal of Computer Science and Technology, 2020
- The M4 Competition: 100,000 time series and 61 forecasting methodsInternational Journal of Forecasting, 2020
- 3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation2016
- Data smoothing and numerical differentiation by a regularization methodComputers & Chemical Engineering, 2010
- Theoretical analysis of batch and on-line training for gradient descent learning in neural networksNeurocomputing, 2009
- Data discretization unificationKnowledge and Information Systems, 2008